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k  relative  cost  of  the  various  candidate  features  of  the  product.  Finally, 
the  user  must  communicate,  verbally  or  in  writing,  the  requirements 
specifications  in  clear,  unambiguous  language. } 

-  - 

The  goal  of  this  investigation  was  to  evaluate  the  capability  of  non¬ 
programming  users  (in  this  case  experienced  inventory  managers)  to 
develop  specifications  for  an  inventory  error-control  system.  Participants 
specified  tests  from  a  set  of  available  tests  to  detect  possible  errors  in 
inventory  change-records,  i.e.,  in  inventory  updates.  Participants  worked 
on  problems  at  various  levels  of  complexity.  For  each  set  of  change- 
record  tests  specified,  which  defined  a  candidate  design,  the  total  system 
cost  was  automatically  calculated  and  fed  back  to -the  participants. 
Participants  attempted  to  specify  the  least-cost  design. 

M  . 

Costing-aids  were  provided  that  were  analogs  of  aids  that  are  ex¬ 
pected  to  be  presently  available.  The  costing-aids  provided  an  increasing 
data  on  system  cost:  the  total  system  cost,  the  total  system  cost  plus 
the  costs  of  each  part  of  the  system,  and  an  automatic  sort  of  previous 
designs  according  to  cost.  Problem-complexity  had  a  strong  effect  on 
performance;  greater  problem-complexity  resulted  in  more  costly  designs. 
Further,  the  more  complete  costing-aids  tended  to  degrade  performance 
on  the  simple  problems,  yet  they  improved  performance  on  the  more 
complex  problem.  Those  effects,  however,  were  not  "statistically 
significant. 

v 

It  was  concluded  that  the  ability  of  non-programming  users  to 
develop  least-cost  requirements  specifications  was  poor  when  using  any 
of  the  aids  that  were  made  available.  The  experimental  results, 
however,  suggested  new  approaches  for  procedural  and  computational 
support  for  the  non-programming  user,  that  could  lead  to  more 
suitable  costing  aids. 
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Abstract 


The  difficulties  of  the  non-programming  user  in  developing  quality 
software-requirements  specifications  are  well  known  and  described 
in  the  literature.  Typically,  the  user  will  work  with  a  software 
expert  to  develop  the  specifications.  During  this  process  the  user 
must  learn  new  terms  and  concepts,  and  must  attempt  to  identify 
the  required  functions  of  the  resulting  software  product.  Further, 
the  user  must  learn  the  relative  cost  of  the  various  candidate 
features  of  the  product.  Finally,  the  user  must  communicate, 
verbally  or  in  writing,  the  requirements  specifications  in  clear, 
unambiguous  language. 

The  goal  of  this  investigation  was  to  evaluate  the  capability  of  non¬ 
programming  users  (in  this  case  experienced  inventory  managers) 
to  develop  specifications  for  an  inventory  error-control  system. 
Participants  specified  tests  from  a  set  of  available  tests  to  detect 
possible  errors  in  inventory  change-records,  i.e.,  in  inventory 
updates.  Participants  worked  on  problems  at  various  levels  of 
complexity.  For  each  set  of  change- record  tests  specified,  which 
defined  a  candidate  design,  the  total  system  cost  was  automatically 
calculated  and  fed  back  to  the  participants.  Participants  attempted 
to  specify  the  least-cost  design. 

Costing-aids  were  provided  that  were  analogs  of  aids  that  are  expected 
to  be  presently  available.  The  costing-aids  provided  an  increasing 
data  on  system  cost:  the  total  system  cost,  the  total  system  cost 
plus  the  costs  of  each  part  of  the  system,  and  an  automatic  sort 
of  previous  designs  according  to  cost.  Problem-complexity  had 
a  strong  effect  on  performance;  greater  problem-complexity  resulted 
in  more  costly  designs.  Further,  the  more  complete  costing-aids 
tended  to  degrade  performance  on  the  simple  problems,  yet  they 
improved  performance  on  the  more  complex  problem.  Those  effects, 
however,  were  not  statistically  significant. 


It  was  concluded  that  the  ability  of  non-programming  users  to  develop 
least-cost  requirements  specifications  was  poor  when  using  any  of 
the  aids  that  were  made  available.  The  experimental  results, 
however,  suggested  new  approaches  for  procedural  and  computational 
support  for  the  non-programming  user,  that  could  lead  to  more 
suitable  costing  aids. 


INTRODUCTION 


Problem  Statement 

Although  errors  in  software  occur  in  all  stages  of  the  software 
life  cycle,  errors  that  occur  early  in  the  cycle,  especially  in  the 
preparation  of  software-requirements  specifications  (RS),  are  critical 
because  they  are  difficult  to  detect.  Also,  the  cost  of  correcting  a 
requirement-specification  error  increases  as  its  detection  is  delayed 
through  subsequent  software  development  states.  Further  complications, 
according  to  Boehm  (1976),  are  that  most  errors  in  software-develop¬ 
ment  occur  in  the  early  design  stages  and  are  frequently  errors  of 
omission. 

Incomplete  or  inaccurate  requirements  specifications  cause 
difficulty  in  other  software  development  stages  as  well:  systematic, 
top-down  design  suffers  from  a  lack  of  complete  specifications;  testing 
is  inadequate  due  to  lack  of  complete,  accurate  requirements  to  test 
against;  and  project  management  suffers  from  the  lack  of  a  complete 
statement  against  which  progress  can  be  measured. 

Many  existing  methodologies  developed  to  improve  software 
quality  unfortunately  apply  only  to  the  design  stages  (both  preliminary 
and  detailed)  and  the  subsequent  stages.  These  methodologies,  which 
are  directed  toward  improving  the  software-development  process  only  after 
RS  have  been  specified,  as  a  result  neglect  the  process  of  producing 
quality  RS,  themselves.  This  imbalance  in  the  distribution  of  software 
methodologies  and  aids,  in  which  software  design  and  testing  are 
favored  over  the  preparation  of  RS,  is  illustrated  by  the  lack  of  a 
requirement-preparation  "block"  in  typical  descriptions  of  the  software 
development  process,  as  shown  in  Figure  1  .  The  process  as  typically 
visualized  begins  with  a  requirements  block,  as  if  to  indicate  that  the 
process  starts  with  RS  already  in  existance.  Improvements  in  the 
requirements-development  process  have  been  generally  limited  to  the 
development  of  notational  methods  for  recording  requirements .  Actually, 
of  course,  the  process  starts,  as  shown  in  Figure  2,  with  user-needs, 
which  may  or  may  not  be  well  understood  by  the  user,  but  from  which 
the  RS  must  be  developed.  It  is  this  actual  process,  the  transformation 
of  vague  user-needs  into  precise  RS,  often  with  the  user  and  a  software 
expert  working  together,  that  is  of  interest  here. 
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System  Requirements 


Figure  1 .  Typically  but  Erroneously  Assumed 
Software  Life  Cycle 
2 


Background 

At  present,  the  tools  available  for  software  development  fall 
into  one  of  two  classes.  The  first  includes  the  well-known  design- 
aids  such  as  Structured  Design  (Yourdon  &  Constantine,  1979)  , 

Jackson's  Method  (Jackson,  1975),  and  Logical  Construction  of  Program 
(Warmer,  1981  and  Orr,  1981).  All  of  these  design  aids  use  RS,  which 
are  assumed  to  be  correct,  as  their  starting  point. 

The  second  class  of  aids  provides  structures  with  which  to 
record  and  analyze  RS.  In  this  latter  class  is  a  system  called  ISDOS 
(Teichroew  &  Hershey,  1979),  which  uses  a  problem-statement  language 
(PSL)  and  a  problem-statement  analyser  (PSA).  ISDOS  permits  a 
formal  description  of  a  system  in  terms  of  entities,  classes,  and  relation¬ 
ships,  and  automatically  provides  analytical  summaries,  such  as  problem- 
statements,  directories,  hierarchical-structure  reports,  and  graphical 
summaries  of  data  flow  and  data  relationships.  Another  example  in  the 
class  is  a  method  called  Software  Requirements  Engineering  Programs 
(SREP),  described  by  Boehm  (1976)  in  a  survey  of  methodologies. 

SREP  uses  the  data-management  system  of  ISDOS  and,  in  addition, 
produces  functional  simulations  from  requirements  statements.  SREP 
is  used  for  configuration  control,  traceability  from  requirements  to 
design,  and  report  generation.  Still  a  further  method  in  the  second 
class.  Structured  Analysis  for  Requirements  Definition,  is  described 
by  Ross  and  Schoman  (1977).  Part  of  it  is  a  Structured  Analysis 
and  Design  Technique  (SADT)  for  analyzing  requirements  using 
graphical  techniques. 

All  these  methods,  despite  their  complexity,  contain  a  common 
limitation:  that  of  providing  a  structure  for  recording  RS  and  then  for 
analyzing  those  RS,  but  not  for  supporting  the  process  of  developing 
RS  from  a  user's  needs. 


In  an  extensive  survey  and  review  of  the  status  of  software- 
requirements  methods,  Ramamoorthy  &  So  (1978)  identified  the  same 
requirements-development  problems  referred  to  above;  namely,  that  a 
large  percentage  of  the  total  errors  in  software  development  occur  in 
the  requirements  specifications,  and  that  these  errors  cause  serious 
problems  leading  to  high  costs,  unresponsive  products,  slippage  of 
production  schedules,  and  difficulty  in  system  operation  and  maintenance. 
They  go  on  to  briefly  describe  a  number  of  methodologies  for  RS 
documentation  which  they  feel  may  aid  in  the  software-design  process. 

Richhart  (1983)  provides  a  list  of  the  manual,  semi-automated 
and  automated  tools  available  for  recording  RS  and  the  related  software- 
design  processes .  These  tools,  some  of  which  have  been  identified  above, 
are  given  in  Appendix  A.  Richhart  also  gives  a  description  of  software- 
development  procedures  used  in  the  Navy  and  the  Air  Force. 


According  to  Howden  (1982),  requirements  specifications  are 
formal,  contractual  documents  that  define  the  system  to  be  built.  RS 
should  thus  be: 

Complete 

Precise  and  Unambiguous 

Machine-Readable  but  Human  Intelligible 

Incrementally  Constructed  and  Reviewed 

Testable  (Validable  Plans) 

Cross-Referenced 

Change  Controlled  (Formal  Reviews) 

User  Accepted 

Problems  with  Requirements  Specifications 

In  contrast  to  the  desired  features  given  by  Howden  (1982), 
Richhart  (1983)  developed  a  list  of  common  problems  with  RS,  given 
here  in  its  entirety: 

1 .  Developers  and  customers  often  interpret  RS 
differently  (different  cultures). 

2.  Users  do  not  know  in  detail  what  they  want  until 
they  use  a  version  of  it. 

3.  Requirements  change  quickly  and  often  frequently 
while  the  system  is  being  developed:  This  includes 
both  uncontrollable,  external  factors  and  internal 
redefinement . 

4.  Specification  document  is  long  and  boring  (not 
fully  read  or  understood  by  users). 

5.  Specifications  are  frozen  when  users  are  only 
halfway  up  the  learning  curve . 

6.  The  specification  document  itself  contains  errors. 
(In  a  typical  case  for  a  large  corporation,  64% 

of  its  bugs  were  in  requirements-analysis  and 
design.  (Martin, 1982)  ) 


7.  The  act  of  providing  what  an  end-user  says  he 
needs  changes  his  perception  of  those  needs . 

(Martin^  1 982) 

8.  APD-systems  staff  usually  take  the  initiative 
and  design  what  they  "thought  the  required 
system  was." 

9.  Different  needs  for  different  users  sometimes  conflict 

10.  Narrative  specifications  must  be  digested  and 
converted  into  a  useful,  non-procedural  model 
to  serve  as  the  basis  for  deriving  the  modules. 

11.  Specifications  are  seldom  complete.  Frequently 
are  20%  i ncomp l ete.  (Nolan,  1981  ) 

12.  The  system  the  user  gets  may  be  exactly  what  he 
asked  for,  but  it  may  not  solve  his  problem  at  all* 
(Zahniser,  1981) 

13.  The  eventual  operational  users  will  probably  not 
be  the  same  ones  who  helped  develop  the  specs 
(due  to  personnel  turnover,  promotion,  etc.), 
and  the  new  users  will  usually  want  something 
different . 

1 4 .  The  fact  is  that  many  of  the  most  important 
potential  users  of  data  processing  do  not  know 
what  they  want  until  they  experience  using  the 
system.  When  they  first  experience  it,  many 
changes  are  needed  to  make  them  comfortable 
with  it  and  to  meet  their  basic  requirements. 

(Martin,  1 982) 

15.  The  requirements  for  management-information 
systems  cannot  be  specified  beforehand ,  and 
almost  every  attempt  to  do  so  has  failed .  The 
requirements  change  as  soon  as  an  executive 
starts  to  use  his  terminal.  (Martin ,1982) 


16.  The  important  problem  is  how  to  migrate  from 
conventional  programming  and  the  traditional  life 
cycle  into  the  development  of  methodologies  that 
are  fast,  flexible,  interactive,  and  employable 
by  end-users;  methodologies  in  which  interactive 
prototyping  replaces  formal,  voluminous  specifica¬ 
tions  which  must  be  frozen;  methodologies  with 
which  end-users  can  create  and  continuously 
modify  their  own  applications .  (Martin,  1982) 

17.  Approaches  which  use  propositional  calculus  and 
formal-definition  languages  clearly  do  not  relate 
to  the  grassroots  user  at  all.  (Zahniser,  1981) 

Solutions  Suggested  for  Solving  RS  Problems 
Martin  (1982) 

1 .  High-level  tools  for  user-driven  specification  and 
development. 

2.  Prototypes  to  largely  replace  the  use  of  lengthly, 
written  requirements  documents.  Application 
generators  for  prototyping.  After  prototyping, 
complete  design  to  achieve  machine  efficiency, 
security,  telecommunications  networking,  data¬ 
base  creation,  etc. 

3.  Use  central  data-base/information-control  systems. 

4.  Highly  interative/interrelated  design  &  development. 
Zahniser  (1961) 

1 .  Build  a  very  high-level,  abstract  view  of  the 
system  in  user  terms,  employing: 


A.  System  surveys,  ( feasibility  studies,  baseline 
descriptions) 

B.  Business  requirements  analysis. 

C.  Structured  problem  definition. 
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2.  Develop  automated  tools  for  working  with  the  user, 
including: 

A.  Interactive  problem-analyser 

B.  User-friendly  data  dictionary 

C.  Cost-benefit  analysis  program 

D.  A  set  of  checklists,  guidelines,  and  heuristics 
to  serve  as  reference  guides  for  measuring 
the  completeness  and  quality  of  the  design 
and  requirements. 

Bearley  and  Wood  (1980) 

The  analyst  must  . . .  "assist  the  user  in  stating  these 
perceived  problems  in  a  clear  and  understandable  manner.  To 
do  this  required  a  form  of  problem  analysis  that  not  only  identifies 
the  user's  perceived  problems ,  but  also  quantifies  reasons  for 
the  problems  and  defines  criteria  by  which  the  user,  analyst 
and  management  can  determine  whether  the  problem  has  been 
resolved." 

Research  on  Development  of  Requirements  Specifications 

Miller  (1978)  investigated  the  interactive  process  between  a 
user  (client)  and  software  designer  in  analyzing  the  user's  needs, 
establishing  RS,  and  developing  a  software  design.  His  description  of 
the  interactive  process  identified  four  steps,  end  is  presented  below. 

For  ease  of  reference  in  what  follows,  we  give  all  four  of  Miller's  steps, 
even  though  our  interest  here  is  limited  to  the  first  two  steps. 

The  four  steps  Miller  identified  are: 

1 .  Problem  understanding,  arriving  at  a  general 
agreement  as  to  what  are: 

a.  the  goal  objectives, 

b.  the  system  or  environments  involved, 

c.  the  constraints  (on  performance,  delivery, 
costs,  etc.), 

d.  the  resources  available  for  system-design 
development. 
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2.  Functional  requirements  specifications  determining 
precisely  what  the  final  product  must  be  like, 
including: 

a.  every  important  aspect  of  the  product's 
internal  performance, 

b.  the  characteristics  of  its  embedded  operator/ 
user  population, 

c.  its  relationship  to  other  systems  and  environments, 
and 

d.  the  development  constraints 

3.  Overall  high-level  design  translating  the  functional 
requirements  into  a  comprehensive  design  which 
specifies  the  major  components  of  the  to-be- 
developed  product,  and  describing  for  each  component: 

a.  the  goals  to  be  achieved  by  the  component, 

b.  the  characteristics  of  all  factors  to  which  the 
component  is  to  be  sensitive,  i.e. ,  the  input, 

c.  the  characteristics  of  the  effects  the  component 
must  achieve,  i.e.,  the  output, 

d.  the  internal  structures  of  the  component,  and 

e.  the  general  principle(s)  of  any  operation 
sequences  within  the  component  information¬ 
processing  procedures. 

4.  Detailed  design  suitable  for  prototype  development. 

The  steps  of  interest  here  are  the  first  and  second,  which  start 
with  the  initial  discussions  of  the  problem  with  the  client  and  end  with 
preparation  of  formal  RS.  We  will  not  treat  the  high-level  or  detailed- 
design  steps. 

Miller  investigated  the  transformation  of  a  client's  vague, 
initial  specifications  into  precise  and  formal  specifications.  He  described, 
in  particular,  the  functions  of  the  client  and  the  software  designer  by 
describing  the  interchange  between  the  two,  and  he  noted  that  designers 
often  use  the  technique  of  suggesting  particular  pieces  of  equipment  or 
procedures  that  might  be  (or  might  at  least  approximate)  an  acceptable 
solution.  The  client,  in  rejecting  some  of  these  suggestions,  modifies 
his  own  requirements  statements.  As  a  result  of  this  designer-client 
interchange,  the  client  clarifies  his  own  understanding  of  the  problem, 
and,by  working  together,the  client  and  designer  arrive  at  an  acceptable 
solution. 


Miller  pointed  out  that  the  role  of  the  designer  is  to  provide 
facts  about  the  real  world,  in  terms  of  the  properties  of  equipment  and 
alternative  solutions,  as  well  as  to  ask  questions  which,  while  providing 
clarification,  frequently  may  have  the  effect  of  inducing  the  client  to 
identify  a  new  problem  or  to  better  conceptualize  the  present  problem. 
Miller  further  identified  a  sequence  of  six  states  which  the  client  and 
designer  use  sequentially.  These  six  states  are: 

1 .  Goal  statement 

2 .  Goal  elaboration 

3.  (Sub)  Solution  outline 

4.  (Sub)  Solution  elaboration 

5.  (Sub)  Solution  explication 

6.  Agreement  on  (Sub)  solution. 

Miller  indicated  that  this  state-sequence  was  used  iteratively,  but  that 
sometimes  the  sequence  was  truncated  in  order  to  start  a  new  sequence 
in  the  pursuit  of  a  different  solution. 

The  results  by  Miller  suggest  that  the  process  of  transforming 
a  user's  needs  into  a  formal  statement  of  requirements  may  benefit  from 
the  interchange  between  a  client  knowledgeable  about  his  own  needs 
and  a  software  designer  knowledgeable  about  the  capabilities  of  computer 
systems.  According  to  this  model,  the  client's  concept  of  his  needs 
grows  as  a  result  of  the  interchange,  and  he  or  she  becomes  aware  of 
new  and  different  possible  solutions  to  the  problem.  New  solutions  evolve 
iteratively  until  a  final  solution  emerges  that  is  accepted  by  both  the 
client  and  designer  as  being  complete  and  feasible. 

The  question  arises,  how  comprehensive  ought  these  interchanges 
to  be  to  evolve  a  feasible  solution?  For  instance,  when  preparing  RS 
for  a  large  computer  system,  it  may  not  be  possible  for  all  the  user- 
clients  to  have  a  useful  interchange  with  one,  or  more,  of  the  designers. 
Not  only  would  there  have  to  be  multiple  client-designer  interchanges, 
but  there  would  also  have  to  be  multiple  interchanges  (discussions  of 
tradeoffs  among  the  many  interests)  among  the  user-clients  themselves, 
and  perhaps  multiple  interchanges  among  the  several  designers.  At 
present,  when  specifications  for  the  development  of  a  large  software 
system  are  considered,  the  user-client  develops  formal  specifications 
without  extensive  interchange  concerning  the  ultimate  designs.  The  RS 
are  then  presented  to  designers,  perhaps  in  the  form  of  a  request-for- 


quotation.  Such  a  procedure,  though  often  used  for  large  software  projects 
is  formal  and  prohibits  the  informal  interchange  described  above  that 
might  well  be  used  to  advantage  in  the  development  of  a  smaller  software 
system . 


Goals  of  this  Research  Program 

Goals  of  PMA's  research  program,  for  which  this  report  is 
the  first  technical  report,  were  to: 

Investigate  the  nature  of  the  user/software-expert 
interaction  in  developing  RS  in  order  to  identify 
factors  that  limit  the  quality  of  the  resulting  RS. 

Design  aids  to  improve  the  quality  of  RS. 


Conduct  experiments  to  test  and  evaluate  the  RS 
quality-improvement  aids. 


METHOD  OF  APPROACH 


The  Series  of  Experiments 

A  series  of  experiments  was  designed  as  a  means  for  better 
understanding  the  process  by  which  individuals  not  skilled  in  software, 
but  skilled  in  at  least  one  software- application  area,  develop  RS  by 
working  with  a  software  expert.  A  further  purpose  of  the  experiments 
was  to  design  and  test  aids  with  which  to  assist  software  users  in  the 
process  of  developing  RS.  The  four  experiments  which  were  planned 
sought  to: 


1 .  Identify  the  capabilities  of,  and  the  strategies 
employed  by,  software  users  in  specifying  a 
minimum-cost  inventory-control  system,  and 
to  establish  a  base-line  study  of  presently 
available  aids. 

2.  Develop  and  evaluate  more  sophisticated  aids  with 
which  to  assist  a  user  in  providing  complete  RS. 

3 .  Develop  and  evaluate  aids  that  a  user  and  software 
expert  could  use  together  to  build  a  working 
vocabulary  of  software  specifics  and  application- 
specific  terms. 

4.  Develop  and  evaluate  aids  to  assist  a  neophyte 
user  to  develop  RS . 

This  first  technical  report  presents  both  a  description  and  the 
results  of  the  first  experiment  cited  above  —  an  investigation  of  the 
capability  of  users  to  develop  RS  for  a  minimum-cost  inventory-control 
system,  including  identification  of  the  strategies  employed  by  users  who 
were  successful  in  developing  minimum-cost  systems.  Since  the 
development  of  RS  by  ah  individual  not  skilled  in  software  requires  a 
dialogue  with  a  software  expert,  several  additional  investigations  are 
currently  planned  (See  2,  3,  and  4  above)  to  evaluate  the  use  of  a  computer 
to  facilitate  that  dialogue. 

Experiment  Task:  An  Inventory-Control  Problem 

The  experiment  task  was  to  develop  software  RS  for  an  inventory- 
control  problem  by  means  of  a  recorded  interaction  between  a  user  and  a 
simulated  software  designer. 


Problem:  Inventory  Control 

An  important  aspect  of  inventory  control  is  the  maintenance  of 
records:  of  present  stock,  of  the  amount  of  stock  on  order,  of  recent 
transactions,  and  of  transaction  histories.  At  periodic  intervals,  daily, 
weekly,  or  monthly,  the  old  master  inventory-file  is  read,  transactions 
are  recorded,  new  stock  levels  are  computed,  new  stock-order  recom¬ 
mendations  and  other  reports  are  written,  and  a  new,  updated,  master 
file  is  produced.  These  periodic  updates  of  the  old  master  file  are 
accomplished  by  means  of  what  are  known  as  change- records .  A 
change-record  is  the  replacement  of  a  datum  in  the  old  master  file 
with  a  new  datum.  When  all  change-records  have  been  entered,  a 
new  master  file  is  produced. 

If  a  change-record  contains  an  error  that  is  not  detected  prior 
to  its  incorporation  in  the  new  master  file,  the  erroneous  change-record 
may  result  in  unnecessary  costs  to  the  organization.  For  instance,  if 
a  change-record  error  indicates  that  stock  of  a  particular  item  is  lower 
than  the  actual  level,  additional,  unnecessary  parts  may  be  ordered. 
Conversely,  if  a  change-record  error  indicates  that  the  stock  of  a 
particular  item  is  higher  than  the  actual  level,  needed  parts  may  not 
be  ordered,  possibly  resulting  in  a  production  slow-down.  Since 
change-record  errors  can  result  in  unnecessary  costs  to  the  organization 
it  is  well  to  consider  specifying  tests  for  these  change-records  in  order 
to  detect  any  errors  before  the  change-records  are  entered  into  the  new 
master  file. 

Tests  of  change-records,  however,  are  not  available  without 
cost.  Tests  must  be  developed,  evaluated,  and  programmed  and 
maintained.  Further,  tests  that  result  in  "false  alarms",  i.e., 
indications  of  an  error  when  one  does  not  exist,  result  in  unnecessary 
error- investigation  costs.  Expensive  tests  that  only  detect  errors  that 
have  little  cost- impact  should  not  be  used,  and  conversely,  tests  that 
detect  errors  with  high  cost-impact  compared  to  the  test  cost  should 
be  used.  Thus,  change-record  tests  should  be  carefully  specified  so  as 
to  minimize  the  total  cost  to  the  organization.  Total  cost  consists  of 
the  cost  of  undetected  change- record  errors  plus  the  cost  of  test 
development  and  maintenance. 

The  Participant's  Task 

The  participant's  task  was  to  specify  a  set  of  change-record 
tests  that  resulted  in  the  least-total-cost  system.  Total  cost  was,  as 
stated  previously,  the  cost  of  undetected  errors  plus  the  cost  (of  develop 
ment  &  maintenance)  of  the  tests  used. 


Since  the  participant,  who  in  all  cases  was  a  manager  experienced 
in  inventory  systems,  was  not  expected  to  have  expert  knowledge  of 
the  type  and  frequency  of  change-record  errors,  the  cost-impact  of  each 
error,  the  cost  of  change-record  tests,  or  the  effectiveness  of  each 
test,  a  simulated  software-expert  was  provided  to  calculate  the  expected 
cost  of  each  of  the  participant's  designs  (the  set  of  tests  he  or  she 
specified)  and  to  feed  this  total-cost  information  back  to  the  participant. 

Fourteen  inventory  records,  shown  in  Table  1,  were  presented 
to  the  participant.  Multiple  types  of  errors,  listed  in  Table  2,  were  allowed 
for  each  record,  giving  a  total  of  44  possible  change-record/change- 
record  error  combinations.  An  error  probability  matrix,  given  in 
Appendix  C,  was  developed  containing  the  probability  for  each  erroi — 
type  for  each  change-record.  The  error  probabilities  selected  were 
based  on  a  search  of  the  literature  on  error-rates  for  vaious  types  of 
tasks.  It  was  found  not  to  be  possible,  however,  to  use  exact  erroi — 
rates,  for  in  many  cases  such  error-rates  were  either  unknown  or 
were  given  in  the  literature  as  a  broad  range  of  values.  Consequently, 
only  representative  erroi — probabilities  were  used. 

The  participant's  task  was  to  select  none,  one,  or  more  than 
one  test  for  each  change- record.  The  maximum  number  of  tests  available, 
shown  in  Table  3,  was  13;  the  number  used  for  each  problem  was  an 
experiment  variable,  described  subsequently.  Each  test  was  supplied  with 
a  name  and  a  brief  description  of  its  function.  The  description  of  a 
test's  function  was  general  and  offered  no  more  than  guidance  for  the 
test's  use.  In  some  instances,  the  inappropriateness  of  a  test  was 
reasonably  obvious .  Thus ,  a  numeric  range  test  was  inappropriate 
for  the  "part  name"  change-record  because  a  name  is  not  a  number. 
Likewise,  the  "alpha  range"  test  was  inappropriate  for  checking  the 
"part  number",  or  "quantity  on  hand"  change-records,  etc.,  because 
quantities  do  not  contain  alphabetic  characters.  In  general,  although 
the  test  names  and  descriptions  offered  some  guidance,  reliable  infor¬ 
mation  was  available  only  by  trying  a  test  in  a  specification  and  then 
observing  the  change  in  total  cost  for  a  change- record.  When  an 
additional  test,  for  example  test  X,  was  specified  for  a  change-record 
without  altering  the  tests  already  specified  for  that  change-record  and 
the  resultant  cost  decreased,  reliable  evidence  was  available  that  test 
X's  use  was  beneficial. 

The  effectiveness  of  each  change-record  test  was  represented 
by  a  matrix  of  probabilities  for  correctly  detecting  each  error-type 
for  each  change- record.  Randomizing  the  effectiveness  for  the  four 
problems  (one  pre-test  problem  and  three  experiment  problems) 
required  four  probability  matrices,  given  in  Appendix  C. 


Table  1 

Types  of  Records 


Identification  Records 

1  .  Part  Number 

2.  Part  Name 

Records  Indicating  Present  State 

3.  Quantity  on  Hand 

4.  Quantity  on  Order 

5.  Percent  Damaged  Received  Damaged  on  Last  Order 
Expected  Delivery  Record 

6.  Quantity  Expected  in  one  Month 
Records  Indicating  Conditions  for  Ordering 

7.  Delivery  Time  when  Ordering  Now 

8.  Quantity  Discount 

9.  Recent  Unit  Price 

Records  of  Historical  Data 


10.  Quantity  Used  to  Date  This  Year 
1 1  .  Quantity  Used  This  Month 

12.  Quantity  Used  Last  Year 

13.  Quantity  Used  Last  Month 

14.  Price  Paid  Last  Year 


Table  2 


Types  of  Error  Assigned  to  Each  Change-Record 


1 .  Part  Number 

1 .  Wrong  Part  Number 

2.  Interchange  of  Digits 

3.  Digit  Missing 

4.  No  Number 

2.  Part  Name 

1  .  Wrong  Part  Name 

2.  Misspelled  Name 

3.  No  Name 

3.  Quantity  on  Hand 

1 .  Random  Error 

2.  Order  not  Recorded 

3.  Stock  Issue  not  Recorded 

4.  Quantity  on  Order 

1 .  Random  Error 

2.  Not  Ordered 

3.  Double  Order 

5.  Percent  Damage  Last  Order 

1 .  Random  Error 

2.  Not  Recorded 

6.  Quantity  Expected  in  One  Month 

1 .  Random  Error 

2.  Quantity  Not  Updated  Since  Last  Entry 

7.  Delivery  Time  When  Ordering  Now 

1 .  Random  Error 

2.  Time  Not  Updated  Since  Last  Entry 

8.  Quantity  Discount 

1 .  Random  Error 

2.  Discount  Not  Changed  From  Last  Entry 


9.  Unit  Price 

1 .  Random  Error 

2.  Digits  Interchanged 

3.  Digit  Missing 

10.  Quantity  Used  Last  Year 
1  .  Random  Error 

2.  Quantity  Not  Changed  Since  Last  Entry 

3.  Digits  Interchanged 

4.  Digits  Missing 

1 1  .  Quantity  Used  This  Month 

1 .  Random  Error 

2.  Quantity  Not  Changed  Since  Last  Entry 

3.  Digits  Interchanged 

4.  Digits  Missing 

12.  Quantity  Used  Last  Year 

1 .  Random  Error 

2.  Quantity  Not  Changed  Since  Last  Entry 

3.  Digits  Interchanged 

4.  Digits  Missing 

13.  Quantity  Used  Last  Month 

1 .  Random  Error 

2.  Quantity  Not  Changed  Since  Last  Entry 

3.  Digits  Interchanged 

4.  Digits  Missing 

14.  Price  Paid  Last  Year 

1 .  Random  Error 

2.  Quantity  Not  Changed  Since  Last  Entry 

3.  Digits  Interchanged 

4.  Digits  Missing 


Range  Tests 


1 .  Numeric  Range  Test 

2.  Alpha  Range  Test 

3.  Fixed  Range  Test 

4.  Range  Test  Based  on  Last  Year's  Experience 

5.  Range  Test  Based  on  Last  Month's  Experience 

Consistency  Tests 

6.  Transaction  Test  (Determine  if  the  Quantity  in  the  New  Master 
File  Equals  that  Quantity  in  the  Old  File  Plus  the  Change) 

7.  Name  Test  for  New  Record  (Does  the  Name  Already  Exist 
in  the  New  Record?  ) 

8.  Number  Test  for  New  Record  (Does  the  Number  Already 
Exist  in  the  Old  Records?  ) 

9.  Name  Test  for  Delete  (Is  There  a  Record  With  That  Name 
in  the  Old  File?) 

10.  Number  Test  for  Delete  (Is  There  a  Record  With  That  Number 
in  the  Old  File?) 

1 1  .  Balance  Test  (Quantity  On  Hand  for  Coordinated  Parts  Equal 
Within  a  Specified  Tolerance.) 

12.  Projected  Usage  Test  (Does  the  Quantity  on  Hand  Plus 
Expected  Delivery  Minus  Shrinkage  Equal  or  Exceed  the 
Expected  Usage  Next  Month?) 

13.  Independent  Varification  of  Data  (Audit  of  Record  Changes). 


The  participant’s  task  was  to  enter  candidate  tests  for  each 
change-record,  request  a  cost  update,  review  the  costs,  and  then 


repeat  these  steps  until  time  was  up  or  until  the  participant  believed 
the  minimum-cost  design  had  been  specified.  Three  control  keys 
were  available,  as  follows: 

Press  F1  to  enter  a  design  (a  set  of  tests) 

•  Computer  response:  "enter  Record  [i.e.,  change- 
record]  ID  number" 

—  participant  entered  ID  number 

•  Computer  response:  "enter  number  of  tests" 

—  participant  entered  the  number  of  tests 

•  Computer  response:  "enter  test  number  1" 

—  participant  entered  the  first  test  number,  from 
1  to  13,  and  continued  entering  test-numbers  until 
the  entire  test  design  had  been  entered. 

Press  F2  to  request  cost  of  design 

•  Computer  response:  a  new  design-number  was 
automatically  assigned  each  time  F^  was  pressed. 
Computer  calculated  &  displayed  colt  of  new  design. 

Press  F  to  view  a  previous  design 

O 

•  Computer  response:  "enter  number  of  desired  design 

—  participant  entered  design  number 

•  Computer  response:  displayed  design  requested. 

Note  that  F  ,  F  ,  and  F  were  special  function  keys 
available  on  the  terminal  Thus,  a  single  key,  F^ ,  F^>  or  F  ,  was 
pressed  to  command  the  desired  function. 
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Calculation  of  Total  Cost 

The  cost  of  each  undetected  error  and  the  cost  of  using  each 
available  test  were  represented  by  elements  in  the  cost  matrices 
documented  in  Appendix  C.  Using  the  cost  and  error-probability 
matrices,  the  total  cost  of  any  design  (set  of  tests)  was  calculated. 

This  total  cost,  as  noted  already  above,  was  the  sum  of  the  cost  of 
the  tests  used  plus  the  cost  to  the  organization  of  the  errors  that 
went  undetected  in  spite  of  the  tests.  This  total  cost  may  be  re¬ 
presented  as  follows: 

Let  T.  =  test  k 
k 

CT,  =  cost  of  test  k 
k 

1^.  _  p  if  test  k  is  assigned  to  change-record  i 

JO  if  test  k  is  not  assigned  to  change-record  i 

The  cost  of  all  the  tests  assigned  to  all  change- records  is  therefore 

given  by: 

OT  =£LCTk  x  Ik.  (.) 

i  k 


Now,  to  calculate  the  cost  to  the  organization  of  all  the  errors  that 
remain  undetected  in  spite  of  all  the  tests  used,  let 


PE.. 

ij 


the  probability  of  error  E..,  where 
i  is  the  change-record  number  (1-14), 
and  j  is  the  error-number  for  change- 
record  i  (note:  the  maximum  value  of  j 
varies  with  i  because  the  number  of  error 
types  per  inventory  record  is  not  constant. 
(See  Table  2)) 


PD  .  =  the  probability  of  failing  to  detect  error  E 


ij 


C.. 

‘J 


=  the  cost  to  the  organization  if  error  E.. 
occurs  and  is  not  detected.  lJ 


From  these  definitions  it  follows  that  the  expected  cost£ C..  of 

failing  to  detect  error  E. .  is  just  J 

ij 


£  C . .  =  PE  x  C..  x  PD.. 
iJ  ij  iJ  tj 


(2) 
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The  total  expected  cost  £  C  of  all  undetected  errors  is  therefore 

£-c  PE.,  x  C..  x  PD..  (3) 

i  J  IJ  IJ  ij 

Finally,  the  total  cost  of  any  design  (set  of  tests)  is  just  the  sum  of 


equations  (1)‘and  (3): 

Total  Cost  = 


E2 

i  j 


PE..xC..x  PD..+Zj2jCT  xl,.  (4) 
ij  U  iJ  I  k  k  ki 


Probability  PD.,  of  Failing  to  Detect  Error  E..:  If  there  are 
multiple  tests  assigned  tb  a  change-record ,  an  error  cSuld  be  detected 


N 


by  more  than  one  test.  Thus,  the  probability  of  error-detection  by  1 
or  more  tests  must  be  determined.  If  there  are  N  tests,  there  are  2 
possible  detect-  or  fail-to-detect  events .  We  can  construct  a  representa¬ 
tion  of  these  events  with  the  symbol  T^  now  used  to  indicate  successful 
detection  of  a  specific  error  by  test  k  and  T  now  used  to  indicate 
unsuccessful  error  detection.  The  construction  is  as  follows: 


Event:  T„  T  T  . . .  T  ;  all  tests  detect  the  error 
12  3  n 


one  test  fails  to 
detect  the  error 


Event:  T„  T„  T  . . .  T  ;  all  tests  fail  to  detect 
12  3  n  *  . 

the  error 

A  method  used  by  Boole  (1854)  to  compute  the  probability  of  a^  event  is 
to  replace  the  logic  variables  with  their  respective  probabilities.  If 
PD^.  is  the  probability  that  test  k  will  detect  error  j  on  change-record  i, 
then  PD  =  1-PD^. .  is  the  probability  that  test  k  will  fail  to  detect 
error  j  on  change-record  i.  Applying  Boole's  method,  the  probability 
that  no  test  will  detect  error  j  on  change-record  i  becomes 


or 


PD. .  =  PD  ..  PDrt..  PDrt . PD  .. 

ij  lij  2ij  3ij  nij 


n 


PD..  =  ‘  (1  -PD  . .). 

ij  k  kiy 


After  accounting  for  the  fact  that  a  participant  can  assign  tests 
arbitrarily,  this  becomes 


PD..=  "  (1-PD  x  I  ) 

ij  k  kij  ki 


(5) 


Total  Cost  of  a  Test  Design  in  Terms  of  Known  Quantities:  Returning 
to  equation  (4)  for  the  total  cost  and  substituting  into  it  the  result  just  ob¬ 
tained  for  PD..,  the  total  cost  finally  becomes 
ij 


Total  Cost  =, 


i  J 


PE  xC  x  n  (1-PD  x  I  ) 


ij  ij 


kij 


w 


CT*  x,« 

i  k 


(6) 


This  total  cost  can  also  be  represented  as  the  sum  of  the  costs 


of  each  error  E„,  that  is: 


Total  Cost  =  Total  Cost  .. 

i  j  lJ 


where 


Total  Cost  . .  =  PE. .  x  C..  x  II  (1-PD,  ..  x,I.  .) 

ij  ij  ij  k  v  kij  ki ' 


CT  xl. 
k  k  ki 


The  four  data  matrices  used  in  the  experiment  are  given  in 
Appendix  C .  They  are: 


cu 


=  the  cost  to  the  organization  of  failing  to 
detect  error  j  on  change-record  i, 


PE..  =  the  probability  that  error  j  on  change- 
J  record  i  will  occur. 


PD  ..  =  the  probability  that  test  k  will  detect 


kij 


error  j  on  change-record  i. 
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^  lw-  ;w  v-  yw  y  y  y»  r 


CT^  =  the  cost  of  using  test  k  on  any  given 
change-record . 


Note  in  Appendix  C  that  there  are  4  matrices  for  PD^, 
labeled  PD  ..1  ...  PD  4,  corresponding  to  the  pre-test  and 
3  experimenV  problems,  Respectively. 


Corrected  Total  Cost:  Differential,  or  Corrected  Total  Cost  (CTC) 
was  used  in  the  data  analysis.  The  CTC  was  equal  to  the  total  cost  given 
above  minus  the  minimum  cost  possible,  obtained  when  the  optimal  set 
of  tests  was  specified. 


Design  of  the  Experiment 

The  experiment  used  a  repeated-measures  Latin  Square  design 
(Plan  #9  cited  in  Winer,  1971 ,  pp.  727-736).  The  design  is  shown  in 
Table  4.  The  factors  investigated  were: 

a.  3  levels  of  problem-complexity,*  where  each  level 
required  a  different  amount  of  effort  to  correctly 
specify  the  problem-solution. 

b.  3  levels  erf' costing-aids*,  where  each  level  required 
that  a  different  amount  of  information  be  provided 
by  the  user  to  correctly  specify  the  solution. 


Referring  now  to  Table  5,  the  numbers  in  column  one  are  the 
numbers  assigned  to  the  participants.  The  numbers  in  columns  two, 
three,  and  four  refer  to  the  problem-number  and  aid-number,  respectively. 
The  numbers  in  column  five  refer  to  the  groups  to  which  the  participants 
were  assigned  and,  in  column  six,  to  the  orderof  problem  presentation. 

Each  participant  was  given: 

1 .  A  description,  via  video  tape,  of  the  experiment- 
task  along  with  an  illustrative  solution. 


2.  A  description,  via  video  tape,  of  the  procedure 
for  entering  data  into  the  computer. 


3.  A  pre-test  problem  —  a  low-complexity 
problem  with  the  Level  2  costing-aid. 


4.  Test  Problem  #X,  Aid  Level  l. 


Measures  of  problem-complexity  and  costing-aid  level  are  discussed 


in  subsequent  sections  of  this  report. 
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Table  4 


Experiment  Design 


A  is  a  level  of  problem-complexity. 

B.  is  a  level  of  costing-aid. 

J 

Each  combination  of  A.  and  B.  is  an 
exper iment  cell .  J 

G.  is  a  group  of  12  participants. 


Table  5 


5.  Test  Problem  #Y,  Aid  Level  m. 

6.  Test  Problem  #Z,  Aid  Level  n. 

Note  that  values  for  X,  Y,  Z,  l,  m,  n  were  taken  from  Table  5  in 
accordance  with  the  participant's  group  number. 


Experiment  Problems:  A  pre-test  and  three  experiment  problems  were 
used.  Problem-complexity  was  controlled  by  the  number  of  tests 
available  for  assignment  to  each  change- record.  In  the  pre-test, 
tests  1  through  4  were  available.  In  experiment  problems  1,2,  and 
3,  tests  1  through  3,  1  through  8,  and  1  through  13,  respectively, 
were  available.  Tables  6  through  9  give  for  each  problem  the  tests 
permitted,  the  number  of  possible  test  combinations*,  the  total  cost 
when  no  tests  were  specified,  the  total  cost  when  the  optimum  tests 
were  specified,  and  the  optimum  tests  and  the  total  costs  for  each 
individual  change-record .  In  all  cases,  a  maximum  of  6  tests  at 
a  time  could  be  specified  for  a  given  change-record.  This  limitation, 
which  was  imposed  to  avoid  overcrowding  of  the  display,  had  little 
real  impact  because  in  all  problems  specification  of  more  than  three 
tests  resulted  in  high  costs. 


m 


Costing-Aids:  Three  levels  of  costing-aids  were  used  in  the  .experiment. 
The  Level  1  costing-aid  consisted  of  only  the  total  cost  of  the  present 
design,  i.e.,  of  the  present  set  of  specified  tests.  With  the  Level  1 
costing-aid,  only  the  total  cost  over  all  change-records,  and  not  for 
individual  change-records,  was  given.  The  terminal  display  presented 
to  the  participant  for  this  costing-aid  is  given  in  Table  10. 


The  Level  2  costing-aid  consisted  of  the  total  cost  of  the 
present  design  plus  the  total  cost  of  each  component  of  the  design, 
i.e.,  of  the  tests  specified  for  each  change-record.  Table  11  shows 
the  terminal  display  for  this  aid. 

The  Level  3  costing-aid  provided  the  same  information  as  the 
Level  2  aid  and,  in  addition,  provtded  a  listing  of  previous  designs, 
ordered  according  to  cost,  for  each  change -record.  Table  12  shows 
the  terminal  display  for  this  aid. 


*  The  number  of  possible  test  combinations  was  given  by  a  sum  of 

binomial  coefficients.  Thus,  if  there  were  M  tests  available  and  a 

maximum  of  N  tests  at  a  time  could  be  chosen,  then  the  total  number 

of  possible  test  combinations  was  N  ....  .  ....  . .  i 

K  /M\  ,  where  /M\  _  M 1  _ 

\n  /  \n  J  n !  M-n ! 


.  %  -V.s 


wo 


Table  6 

Pre-Test  Problem 


1.  Tests  permitted 

2.  Number  of  possible  test 
combinations 

3.  Total  cost  when  no  tests 
were  specified 

4.  Total  cost  when  optimum 
tests  were  specified  for 
each  change-record 


1,  2,  3,  4 
16  * 

$18,465.00 
$  6,960.23 


Change- 

Record 

Number 


Optimum  Set 
of  Tests 


C  hang  e-R  ec  ord 
Total  Cost  for 
Optimum  Tests 


Table  f 


Experiment  Problem-Complexity  Level  1 


1 . 

Tests  permitted 

1,  2,  3 

2. 

Number  of  possible 

test  combinations 

8* 

3. 

Total  cost  when  no  tests 

were  specified 

$18,465.00 

4. 

Total  cost  when  optimum 
tests  were  specified  for 

each  change-record 

$  9,300.28 

Change- 

O 

CD 

Record 

Optimum  Set 

Total  Cost 

Number 

of  Tests 

Optimum  1 

1 

3 

771.85 

2 

2,3 

428.84 

3 

1 

1595.00 

4 

1 

1 100.00 

5 

3 

445.00 

6 

3 

418.38 

7 

1 

342.50 

8 

0 

97.50 

9 

3 

456.25 

10 

1,3 

682.03 

11 

1,3 

769.22 

12 

1 

£30.50 

13 

1,3 

390.06 

14 

1,3 

1173.15 

*  Given  by 


n=0 


3  sum  of  binomial  coefficients 


Table  8 


Experiment  Problem-Complexity  Level  2 


1  . 

Tests  Permitted 

1  through  8  (no  more 
than  6  at  a  time) 

2. 

Number  of  possible  test 
combinations 

247* 

3. 

Total  cost  when  no  tests 
were  specified 

$18,465.00 

4. 

Total  cost  when  optimum 
tests  were  specified  for 
each  change-record 

$6,503.62 

Change- 

Change-Record 

Record 

Optimum  Set 

Total  Cost 

Number 

of  Tests 

for  Optimum  Tests 

1 

8 

211.00 

2 

5 

387 . 25 

3 

3,4 

1 ,377.87 

4 

3,7 

381.50 

5 

3 

175.00 

6 

3,8 

256.75 

7 

3,8 

372.50 

8 

0 

97.50 

9 

1 

263.30 

10 

4,7 

429.46 

11 

3,7 

62.28 

12 

3 

395.65 

13 

4 

337 . 1 5 

14 

4,5 

800.85 

Given  by  6 

i  ,  a  sum  of  binomial  coefficients,  limited  to  6  tests 

1 

* ir* r 


urt-rvr c»  ui 


Table  9 


Experiment  Problem-Complexity  Level  3 


1 . 

T ests  Permitted 

1  through  13  (no  more 

than  6  at  a  time) 

2. 

Number  of  possible 

4668* 

test  combinations 

3. 

Total  costs  when  no  tests 

$18,465.00 

were  specified 

4. 

Total  cost  when  optimum 

$7,855.49 

tests  were  specified  for  each 

change-record 

Change- 

Change -Record 

Record 

Optimum  Set 

Total  Cost 

Number 

of  Tests 

for  Optimum  Tests 

1 

8 

331 .30 

2 

5,7 

453.53 

3 

4,11 

1 ,207.50 

4 

3 

1,135.00 

5 

3,7 

365.00 

6 

3,7 

359.96 

7 

3 

520.00 

8 

0 

97.50 

9 

3,5 

428 . 59 

10 

3,4 

625.06 

11 

7,8,9 

352.28 

12 

1,3 

772.07 

13 

7,8,9 

352.28 

1  A 

3.4 

855.37 

Given  by 


a  sum  of  binomial  coefficients,  limited  to  6  tests 


Change-Record  Field 


Part  Number 

Part  Name 

Quantity  on  Hand 

Quantity  on  Order 

Percent  Damaged  in  Last  Order 

Quantity  Expected  in  1  Month 

Delivery  Time  When  Ordering  Now 

Quantity  Discount 

Unit  Price 

Quantity  Used  This  Year 
Quantity  Used  This  Month 
Quantity  Used  Last  Year 
Quantity  Used  Last  Month 
Price  Paid  Last  Year 


Total  Cost  is  $6,960.23 


Tests 


Record 


Change-Record  Field 

Cost 

Tests 

1. 

Part  Number 

367.38 

4 

0 

C 

0  0  0 

2. 

Part  Name 

545 . 24 

2 

4 

0 

0  0  0 

3. 

Quantity  on  Hand 

1397.50 

3 

0 

0 

0  0  0 

4. 

Quantity  on  Order 

505.00 

4 

0 

0 

0  0  0 

5. 

Percent  Damaged  in  Last  Order 

385.55 

3 

4 

0 

0  0  0 

6. 

Quantity  Expected  in  1  Month 

264.10 

3 

0 

0 

0  0  0 

7. 

Delivery  Time  When  Ordering  Now 

184.75 

3 

0 

0 

0  0  0 

8. 

Quantity  Discount 

97.50 

0 

0 

0 

0  0  0 

9. 

Unit  Price 

349.70 

1 

0 

0 

0  0  0 

10. 

Quantity  Used  This  Year 

453.81 

3 

4 

0 

0  0  0 

11. 

Quantity  Used  This  Month 

1102.25 

1 

0 

0 

0  0  0 

12. 

Quantity  Used  Last  Year 

527.95 

3 

0 

0 

0  0  0 

13. 

Quantity  Used  Last  Month 

190.15 

3 

0 

0 

0  0  0 

14. 

Price  Paid  Last  Year 

589.35 

1 

3 

0 

0  0  0 

Total  Cost  is  $  6,960.23 


Table  12 


9 


g  - 


$  & 


! 


89 


89 


V, 
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Costing-Aid  Level  3  Display 


Change-Record  Field 


Record 

Cost 


Tests 


m 

i. 

Part  Number 

2. 

Part  Name 

3. 

Quantify  on  Hand 

4. 

Quantify  on  Order 

5. 

Percent  Damaged  in  Last  Order 

V 

%* 

6. 

Quantify  Expected  in  1  Month 

'*/ 

7. 

Delivery  Time  When  Ordering  Now 

8. 

Quantify  Discount 

m  *. 

• 

•  \ 

9. 

Unit  Price 

* 

10. 

Quantify  Used  This  Year 

11. 

Quantify  Used  This  Month 

12. 

Quantity  Used  Last  Year 

w5 

13. 

Quantify  Used  Last  Month 

i 

14. 

Price  Paid  Last  Year 

1 

Total  Cost 

367.38 

545.24 

1397.50 

505.00 

385.55 

264.10 

184.75 

97.50 

349.70 

453.81 

1102.25 

527.95 

190.15 

589.35 


4 

2 

3 

4 
3 
3 
3 
0 
1 
3 
1 
3 
3 
1 


$  6f960.23 


Computer  Prompt:  "Enter  Record  ID  Number  for  Extended 

Analysis" 


Design  Number  41  Cost  is  1303.15 


Remember  the  best  Design  will  consist  of  only  the 
best  Records". 


Participant  enters  number:  4 

Computer  Responds:  "Least  Cost  Designs  for  Record  4  are: 

Design  Number  37  Cost  is  505.00 
Design  Number  13  Cost  is  819.00 
Design  Number  8  Cost  is  1102.23 
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0 

0 

0 

0 

0 

o 

0 

0 

0 

0 

0 

0 

0 

0 


Measures 


Performance  Measures 


For  each  of  the  four  experiment  problems  there  was  an 
optimal  set  of  tests  for  each  change-record  —  optimal  in  the 
sense  that  specification  of  those  tests  resulted  in  the  leas4-  total 
cost.  One  performance  measure,  termed  the  "cost  measure", 
was  the  actual  cost  achieved .  Another  measure  was  the  total  cost 
achieved  minus  the  least  total  cost  possible  for  that  problem,  i.e., 
what  was  defined  earlier  in  the  section  on  "Calculation  of  Total 
Cost"  as  the  Corrected  Total  Cost,  or  CTC.  The  least  total  cost 
for  each  problem  is  given  in  Tables  6  through  9. 

Strategy  Measures 

The  following  measures  were  developed  in  an  attempt  to 
capture  the  strategy  used  by  the  participants. 

1 .  Number  of  Designs 

Equal  to  the  total  number  of  designs  (sets  of  tests) 
specified  for  the  problem.  Each  design  could  involve 
specification  of  multiple  tests  for  all  change-records . 
Each  new  design  could  also  involve  modifications 
of  the  previous  tests  for  one  or  more  change-records . 

2.  Time 

Equal  to  the  time,  in  minutes,  used  for  the  pro¬ 
blem.  Maximum  time  permitted  was  60  minutes. 

3.  Change-Record  Selection 


k 


S;  >; 

■k  k 

B  B 


The  purpose  of  this  measure  was  to  determine 
whether  the  participant  tended  to  work  on  the  most- 
costly  change-record  when  specifying  a  set  of  tests. 
Before  each  set  of  tests  was  analyzed,  the  total  cost 
of  each  change-record  was  determined,  and  the  results 
were  ordered  with  the  most-costly  first.  Then,  for 
each  new  set  of  tests  specified  for  change-record  X, 
for  example,  a  partial  score  was  developed  according 


to  the  ordering  of  change- record  X.  If  change-record 
X  had  the  greatest  cost  prior  to  specification  of  trie  new 
set  of  tests  (i.e.,  order  #1),  then  the  partial  score 
was  1.  If  the  change-record  ordering  was  "10",  the 
partial  score  was  a  10,  etc. 

These  partial  scores  were  summed  as  each  new 
set  of  tests  was  specified.  A  low  value  of  the  sum 
indicated  that  the  participant  tended  to  concentrate 
his  or  her  work  on  the  most-costly  change-record, 
where  as  a  high  score  indicated  that  the  participant 
tended  to  work  on  change-records  with  less-than- 
the-greatest  cost.  This  strategy  measure  was  termed 
"Change-Record  Selection"  (CRS). 

Average  Change-Record  Selection 

The  sum  value,  described  above,  developed  as 
the  Change-Record  Selection  strategy-measure  was 
normalized  by  dividing  by  the  total  number  of  test- 
set  modifications  entered  by  the  participant.  This 
normalized  strategy-measure  was  termed  the  "Average 
Change-Record  Selection"  (ACRS). 

Number  of  Designs  Viewed 

The  number  of  designs  viewed  was  the  total 
number  of  times  a  previous  design  was  requested. 

Average  Number  of  Test-Sets  per  Change-Record 


The  number  of  test-sets  per  change-record 
specified  by  the  participant  was  determined.  Then 
the  average  of  those  values  over  the  14  change- 
records  was  computed  and  used  for  a  strategy-measure. 

Total  Number  of  Test-Sets 

The  total  number  of  test-sets  specified  by  the 
participant  for  all  change-records  was  another 
strategy-measu  re . 


8.  Consistency 


This  measure  was  developed  to  ascertain  the 
tendency  of  the  participant  to  work  on  one  change-record 
at  a  time,  i.e.,  by  specifying  and  evaluating  various 
test-sets  for  one  change-record  before  preceeding  to 
the  next  change-record .  This  strategy-measure,  termed 
"Consistency",  developed  a  partial  score  by  giving  1 
point  when  a  test-set  was  first  specified  for  a  change- 
record,  2  points  for  the  second  test-set  (if  different 
from  the  first)  for  that  same  change-record,  4  points 
for  the  third  test-set  (if  different  from  the  first  and 
second)  for  that  same  change-record,  8  points  for  the 
fourth  test— set  ...  etc.  When  a  test— set  was  specified 
for  a  change-record  different  from  the  previous  change- 
record,  the  partial  score  was  reset  to  1 . 

This  consistency  measure  had  a  large  value  when 
the  participant  tended  to  specify,  and  presumably  to 
evaluate,  multiple  test-sets  for  a  given  change-record . 
Conversely,  if  the  participant  tended  to  skip  from  one 
change-record  to  another,  the  measure  value  was  low. 


Participants 

The  participants  were  individuals  experienced  in  some  type  of 
inventory-control  problem  but  were  not  experienced  software  analysts 
or  programmers.  All  participants  were  obtained  via  the  newspaper 
ad  reproduced  in  Appendix  D.  The  participant  demographics  were 
as  follows: 

1.  All  participants  were  high  school  graduates. 

2.  There  were  10  female  participants,  with  ages  ranging 
from  25  years  to  47  years. 

3.  The  average  female  participant’s  age  was  33  years. 


4.  There  were  26  male  participants,  with  ages  ranging 
from  22  years  to  62  years. 


5.  The  average  male  participant's  age  was  33  years. 

6.  The  range  of  years  of  higher  education  was 
0  years  to  8  years. 

7.  The  average  number  of  years  of  higher  education 
was  5  years. 

8.  The  educational  majors  included:  Management  (9), 
Administration  (5),  Business  (5),  Marketing  (3),  and 
Engineering  (2). 

9.  The  work  experience  of  the  participants  included: 
department  stores  (6);  small  businesses  (6);  consulting  (5); 
sales,  management  and  production  analysts  (4);  Armed 
Forces  (4  )j  sales  (3);  fast  food  chains  (3);  and 
supervision  (3). 

Procedure 

Participants  were  scheduled  for  either  a  morning  session, 
beginning  at  8:00  a.m.,  or  an  afternoon  session, beginning  at  1:00  p.m. 
When  a  participant  arrived,  he/she  was  asked  to  fill  out  a  bio¬ 
graphical  questionnaire  (the  questionnaire  is  given  in  Appendix  E) 
to  verify  that  the  participant's  experience  satisfied  the  experiment 
entrance  criteria  and  to  obtain  additional  information  regarding  the 
level  of  experience  in  his/her  particular  field.  If  the  participant's 
experience  did  not  satisfy  the  criteria,  he/she  was  not  used  in  the 
experiment.  If  the  participant's  experience  satified  the  experiment 
entrance  criteria,  the  experiment  was  briefly  explained,  and  the 
participant  was  provided  with  a  consent  form  (Appendix  E), 
having  been  assured  that  no  personal  risk  was  involved.  The 
participant  then  signed  this  form  to  indicate  that  he/she  understood 
these  arrangements . 

The  participant  was  next  seated  in  the  experiment  room. 

The  room  was  approximately  12  x  16  feet  in  size,  with  a  video 
tape  recorder  and  video  monitor  located  on  one  table,  and  a 
computer  and  terminal  on  a  separate  table.  Participants  were 
asked  to  make  themselves  comfortable  and  to  adjust  the  light  and 
ventilation  to  their  satisfaction. 
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Instructions  for  the  experiment  were  presented  in  two  parts, 
both  of  which  were  on  video  tape.  The  first  part  described  the 
experiment  problem  and  gave  a  method  for  solving  the  problem, 
including  an  example-solution .  The  second  part  described  how  to 
enter  data  into  the  computer  and  also  included  an  illustrative 
problem-solution.  Since  this  second  portion  of  the  instructions 
employed  a  dynamic  display  of  the  operation  of  the  computer,  it 
cannot  be  presented  here. 

After  the  instructions  were  presented,  the  participant 
was  seated  in  front  of  the  computer  terminal.  He/she  was  asked 
to  use  the  numbered  keys  labeled  0-9  and  the  RETURN  key, 
as  well  as  three  special-function  keys.  In  addition,  a  pad  of 
paper  and  pencils  were  provided  for  taking  notes.  These  sheets 
were  kept  in  each  participant's  file  for  reference. 

Participants  were  told  that  up  to  one  hour  was  allotted 
for  each  problem,  and  that  the  computer  would  automatically  stop 
the  problem  when  the  time  limit  was  reached.  Participants  were 
permitted  to.  take  a  short  break  between  test,  problems  if  they  desired. 


Data  Collection 


Problem-solutions  were  entered  into  the  computer  by  thu 
participants.  The  computer  recorded  all  keystrokes,  as  well  as 
the  time  of  execution  of  each.  This  method  of  collecting  data 
allowed  a  printout  to  be  generated  listing  detailed  information 
about  each  participant's  activities  during  the  experiment. 


RESULTS 
Mean  Score 


The  mean  scores  for  the  nine  experiment  cells  are  pre¬ 
sented  in  Figure  3.  The  scores  are  Corrected  Total  Cost  (CTC), 
i.e.,  the  total  cost  minus  the  least  possible  cost  for  the  problem. 
These  data  show  a  direct  increase  in  CTC  with  an  increase  in 
problem-complexity.  Further,  the  data  show  an  increase  in  CTC 
with  increasing  costing-aid  level  for  problem-complexity  1  &  2,  but 
show  a  reverse  trend  for  problem  level  3,  the  most  complex  problem. 
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Analysis  #1:  Test  of  Variance  Horn 


Score  variances  over  the  three  levels  of  problem- 
complexity  and  the  three  levels  of  costing-aids  are  given  in  Table  13. 
Since  the  variances  differed,  especially  as  problem-complexity  was 
varied,  tests  of  variance  homogeniety  were  used  to  test  the 
hypothesis  of  equal  variances.  The  Burr-Foster  Q-test  and 
Bartlett's  test  (Anderson,  McLean  1974)  were  used. 

Results .  The  Burr-Foster  Q-test  uses  the  statistic 

4  4  2  2  2 

q  =  (S1  +  ...  +Sp  )  /  (Si  +  ...+Sp  ) 

2 

where  S.  is  the  level  i  variance  and  p  is  the  number  of  levels 
for  the  problem  at  hand.  In  this  experiment,  p  =  3. 

The  statistic  for  the  three  levels  of  problem-complexity  was 

q  =  .3786, 

and  for  the  three  levels  of  costing-aids  was 
q  =  . 3363 . 

Critical  values  of  q  from  the  Q  table  (Anderson,  McLean  1974) 
are  the  following. 

crit  (P=3,  df=20,  »  =  .01)  =  .512 

crit  (P=3 ,  df=20,  a  =  .001)  =  .596 

crit  (P=3,  df=60,  ®  =  .01)  =  .367 


crit  (P=3,  df=60,  » =  .001)=  .384 

Note  that  these  critical  values  depend  on  the  degrees  of  freedom. 

In  this  experiment  cP=35  (=36-1).  Based  on  a  straight-line  inter¬ 
polation: 

crit  (P=3,  df=35,  °  =  .01)  =  .397 

Since  q  for  both  treatments  was  less  than  .397,  there  is  no  reason 
to  believe  that  the  variance  was  not  homogeneous. 
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Table  13 


3 


8,940,535 


Since  interpolation  over  an  extended  range,  as  above, 
can  be  risky,  a  Bartlett  test  was  also  used  to  test  the  equal  - 
variance  hypothesis. 

The  Bartlett  test  (Anderson,  et  al.)  uses  the  statistic: 


where 


X  =  M/C 

M  =  2.3026  (df)  (K  log  S2  -  2  log  S2) 


C  = 


3  df  K 


and  where 


qf  =  the  degrees  of  freedom  per  variance 

K  =  number  of  levels 

2 

S  =  variance  at  each  level 

2 

S  =  average  variance  over  the  K  levels 

For  the  treatment  with  greatest  range  of  variance  (the  three  levels 
of  problem-complexity) 

K  =3 

df  =  2  (=  the  number  of  levels  of  problem-complexity 
minus  1) 

log  S2  =  6.723 
2 

2  log  S  =  20.063 


and,  thus. 


=  8.531 


Critical  values  from  the  X  tables  for  df  =  2  are: 


X  (df=2,  o  <.001)  =  13.82 


(df=2,  a  <.01) 


9.21 


As  with  the  Burr-Foster  Q  test,  there  was  no  reason  to  re 
ject  the  hypothesis  that  the  variances  were  equal. 
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Analysis  #2:  Analysis  of  Variance 


Purpose.  To  determine  whether  problem-complexity,  the 
costing-aids,  or  participant  group  significantly  affected  CTC. 

Method .  Analysis  of  variance. 

Results.  Table  14  presents  the  results  of  the  analysis  of 
variance.  These  results  indicate  that  problem-complexity  was 
highly  significant  and  that  the  group  factor  was  also  significant. 
The  costing-aids  and  the  interaction  effects  were  not  significant. 

Analysis  #3:  Student-Newman-Keuls  Test 


Purpose.  To  determine  whether  the  experiment  factor 
levels  and/or  the  experiment  cells  produced  statistically  different 
results . 

Method .  A  Student-Newman-Keuls  (SNK)  (Sokal,  Rohlf  1969) 
posterior  comparison-of-means  test  was  used  to  test  for  significance 
of  the  effects  of  the  factor  levels  and  experiment  cells. 

Results .  Table  15  presents  the  results  of  the  SNK  test 
applied  to  the  three  levels  of  the  two  experiment  factors.  The 
results  show  that  the  effect  of  the  first  problem-complexity  level 
was  significantly  different  from  that  of  the  second  and  third  levels, 
but  that  the  level-two  and  level-three  effects  were  not  significantly 
different.  These  results  also  show  that  the  effects  of  the  costing- 
aid  levels  were  not  significantly  different. 


Table  16  provides  the  results  of  the  SNK  test  applied  to 
the  nine  experiment  cells.  These  results  show  that  the  first  level 
of  problem-complexity  in  combination  with  any  level  of  costing-aid 
had  effects  that  were  significantly  different  from  those  of  the  third 
problem-complexity  level  combined  with  the  third  level  of  costing- 
aid,  as  well  as  from  the  second  problem-complexity  level  combined 
with  the  third  level  of  costing-aid. 

Table  17  provides  the  results  the  SNK  test  applied  to  the 
three  participant  groups.  The  results  do  not  provide  any  evidence 
that  corresponding  cells  differed  significantly  according  to  group. 


Source  of 
Variation 


Problem-Complexity 

Costing -Aids 

Group 

Residue 

Within  Cell 


299.63 

2 

144.81 

27.28 

7.57 

2 

3.78 

.71 

29.93 

2 

14.96 

2.82 

7.85 

2 

3.92 

.74 

525 . 50 

99 

5.30 

Total 


860.48 


Table  15 


Costing- 

Aid 

Level 
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SNK  Test  Applied  to  Levels  of  Experiment  Factors 


Problem-Complexity  Level 

t 

1st 

2nd 

Level 

1  Level  2 

Level  3 

MEAN 

Difference 

Difference' 

Level  1 

.790 

3.375 

5.740 

3.30 

0.,2ns 

Level  2 

1  .529 

3.550 

5.210 

3.42 

0.48NS 

0.66NS 

t 

Level  3 

1  .629 

5.490 

4.620 

3.90 

' 

Mean 

1  .31 

4.18 

5.19 

« 

• 

' 

1 

1st 

difference 

2 

.88**  1.01 

NS 

** 

« 

2nd 

difference 

3.89 

Least  Significant  Range  (LSR)  for  2 

Means 

=  2.258 

LSR  for  3  Means  =  2 . 563 


Table  16 


SNK  Test  Applied  to  Experiment  Cells 


Problem- 

Costing- 

Mean 

Complexity 

Aid 

Level 

Level 

1 

1 

.790 

.790 

1 

2 

1  .529 

NS  1 . 529 

1 

3 

1  .629 

NS  NS 

1  .629 

2 

1 

3.375 

NS  NS 

NS 

3.375 

2 

2 

3.550 

NS  NS 

NS 

NS 

3 

3 

4.620*1 

*  * 

* 

NS 

3 

2 

5.210  1 

*  * 

* 

NS 

2 

3 

5.490  [ 

*  * 

* 

NS 

3 

1 

5.740  \ 

*  * 

* 

NS 

For  P  <  .01, 

the  least 

significant  range 

LSR  for  9  Means  = 

3.09, 

LSR  for 

8  Means 

= 

3.04 

LSR  for 

7  Means 

= 

2.97 

LSR  for 

6  Means 

= 

2.89 

LSR  for 

5  Means 

— 

2.80 

*  Significantly  different 

from  cell  at  top 

of  column  for  P  < 

.01  level. 

Group 

Mean 

1st  Difference 

2nd 

1 

4.253 

,85NS 

3 

3.402 

NS 

.43 

2 

2.968 

2nd  Difference 


1  .28 


NS 


For  P  <  .01,  the  least  significant  range  (LSR)  for  2  means  =  2. 
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LSR  for  3  Means  =  2.563 


Analysis  #4:  Correlation/Regression  Analyses  of  Experiment  Factors 


Purpose.  To  determine  the  correlations  among  the 
experiment  variables  and  the  ability  of  the  experiment  factors  to 
predict  CTC . 

Method.  Correlation  analysis  among  the  experiment  variables 
and  with  the  CTC  score.  In  addition,  univariate  and  step-wise 
linear  multivariate  regressions  with  the  experiment  variables. 

Results .  Correlations  among  the  experiment  (independent) 
variables  are  given,  along  with  strategy  factors,  in  Table  22 
(Analyses  6).  Tables  18  and  19  present  the  univariate  and  multi¬ 
variate  regressions  using  CTC  as  the  dependent  variable.  The 
univariate  regression  showed  that  only  pre-test  score  and  problem- 
complexity  were  statistically  significant,  explaining  21%  and  30% 
of  the  variance,  respectively. 

Likewise,  the  multivariate  regression  analysis  revealed  that 
pre-test  score,  problem-complexity  and  the  number  of  designs 
specified  by  the  participant  were  significant.  In  neither  the  univariate 
nor  the  multivariate  analyses  were  the  costing-aids  found  to  be 
significant. 

Analysis  #5:  Correlation/Regression  Analyses  of  Demographic  Factors 

Purpose.  To  determine  the  correlations  among  the  demo¬ 
graphic  factors  and  the  ability  of  the  demographic  factors  to  predict 
CTC. 


Method .  Correlation  analysis  among  the  demographic  factors 
and  between  the  demographic  factors  and  CTC.  In  addition,  univariate 
and  step-wise  linear  regression  analyses  with  the  demographic  factors 
as  the  independent  variables  and  CTC  as  the  dependent  variable. 

Results.  Table  20  gives  the  results  of  the  correlation 
analysis  among  the  demographic  factors.  The  correlations  are 
moderate,  thus  allowing  any  combination  of  factors  as  independent 
variables  in  multivariate  regressions.  Table  21  presents  the  results 
of  the  univariate  regression  analysis.  Only  two  factors,  "years- 
of-higher-education"  and  "an  Economics  Degree",  were  found  to  be 
significant,  explaining  18%  and  11%  of  the  CTC  variance,  respectively. 


Table  19 

Multivariate  Regression 

Effects  of  Experiment  Factors 
Dependent  Variable:  CTC 


Regression 

Independent 

Variable 

"  "  Variance 

Coefficient  Explained 

t-Value 

Score  on  Pre-test 
Problem-Complex  ity 


23.75 

.61 


57.2 % 


Source 


Regression 

Residue 


Analysis  of  Variance  Table 
Sum  Sa  DF  Mean  Sa  F-Ratio 


50514.4  3  16838.1  46.32 

37798.4  104  363.4 


Table  21 


Correlation  Analysis  and  Univariate  Regression 
Effect  of  Demographic  Factors  on  CTC 


Regression 


Independent 

Variable 

Correlation 

With  CTC 

r  — 

Coefficient 

Variance 

Explained 

t- Value 

P 

Subject  Age 

.150 

.363 

2.2% 

.884 

NS 

Years  of  Higher 
Education 

.431 

4.272 

18.6% 

2.788 

.01 

Business  Degree 

.238 

10.867 

5.7% 

1  .429 

Engineering  Degree 

-.188 

-14.936 

3.5% 

-1.118 

Education  [Degree 

-.206 

-27 . 522 

4.3% 

-1 .229 

Economics  Degree 

.343 

32.814 

11.8% 

2.128 

.05 

Psychology  Degree 

.190 

18.202 

3.6% 

1.130 

Other  Degree 

.078 

4.087 

(0 

• 

.453 

No.  of  Programming 
Lang .  Known 

-.232 

-  3.996 

5.4% 

-1 .391 

No.  of  Programs 
Written 

-.028 

-  .036 

.1% 

-  .162 

Months  Exper .  in 
Data  Processing 

-.170 

-  .062 

2.9%  ' 

-1 .007 

Years  Exper .  in 
Computer  Science 

.074 

.312 

.6% 

.434 

NS 

Analysis  #6:  Correlation/Regression  Analyses  of  Stratei 
Measures  and  Experiment  Factors 


Purpose.  To  determine  the  correlations  among  the  strategy 
measures  and  the  experiment  factors,  and  the  ability  of  the  strategy 
measures  and  the  experiment  factors  to  predict  CTC. 

Method .  Correlation  analysis  among  the  strategy  measures 
and  the  experiment  factors,  and  also  between  those  measures  and  factors 
and  CTC.  In  addition,  univariate  and  step-wise  linear  regression 
analyses  with  the  strategy  measures  and  experiment  factors  as 
independent  variables  and  CTC  as  the  dependent  variable. 

Result.  Table  22  gives  the  results  of  the  correlation 
analysis.  The  factors  "number  of  designs",  "average  number  of 
test-sets  per  change-record"  and  "total  number  of  test-sets" 
were  highly  correlated  and  therefore  could  not  be  used  together 
as  independent  variables  in  multivariate  regressions.  Tables  23 
and  24  present  the  results  of  the  univariate  and  multivariate 
regression  analyses,  respectively.  In  the  univariate  analysis, 
"problem-complexity",  "pre-test  score",  "time",  "average  change- 
record  selection"  and  "consistency"  were  significant.  In  the 
multivariate  regressions,  the  factors  noted  above  plus  "change- 
record  selection"  were  found  to  be  significant  predictors  of  CTC. 


Table  23 


Correlation  Analysis  and  Univariate  Regression 
Effects  of  Experiment  and  Strategy  Factors  on  CTC 


Regression 

Independent 

Correlation 

C  Variance 

Variable 

With  CTC 

Coefficient  Explained  t-Value 

Session  No. 

-.05 

-1.981 

.3% 

-  .58 

N: 

Problem  Complexity 

.55 

19.389 

30.7% 

6.84 

.oc 

Cost 

.08 

3.069 

.8% 

.90 

NS 

No.  of  Designs 

-.17 

-  .142 

3.0% 

-1.79 

NS 

Time 

.25 

.817 

6.6% 

2.73 

.01 

Change-Record  Selection 

-.09 

-  .766 

CO 

• 

-0.95 

NS 

Ave.  Change-Record  Selection 

.15 

2.51 

2.5% 

1.66 

NS 

No.  of  Designs  Viewed 

.08 

.183 

.6% 

CO 

CO 

• 

NS 

Ave.  No.  of  Test-Sets 

Per  Change-Record 

.02 

.345 

.1% 

.06 

NS 

Total  No.  of  Test- Sets 

.02 

.025 

’  .1% 

.024 

NS 

Consistency 

.29 

.021 

8.9% 

3.22 

.00 

Pre-Test  Score 

.46 

.610 

21  .9% 

5.45 

.00 

l* 
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DISCUSSION 


Effect  of  Costing-Aids  on  Performance 

The  costing-aids  used  in  the  experiment  did  not  result  in  a 
statistically  significant  effect  on  a  participant's  ability  to  identify  the 
least-cost  system.  But  these  costing-aids  were  selected  only  as 
models  of  the  information  typically  available  when  a  non-programming 
user  attempts  to  develop  RS  while  working  with  a  software  expert. 
Costing-aid  Level  1  corresponded  to  the  case  in  which  only  the 
total  cost  of  a  software  product  is  made  available  even  though  there 
are  identifiable  parts  to  the  product  whose  specifications  are  manip¬ 
ulated  by  users  in  an  attempt  to  identify  the  least-cost  RS.  When 
the  cost  of  individual  parts  of  the  product  are  not  available,  the 
user  cannot  direct  his  or  her  efforts  to  the  most-costly  parts.  Such 
a  user  could,  with  bad  luck,  work  on  a  part  that  has  little  cost- 
impact  and  thus  neglect  more  fruitful  areas. 


Costing-aid  Level  2  provided  both  the  product's  total  cost 
and  the  total  cost  of  each  component.  With  this  additional  information, 
the  participant  could  work  on  the  most-costly  items  first  before 
proceeding  to  the  less-costly  items.  Although  this  strategy  did  not 
guarantee  that  the  result  would  be  the  least-cost  product,  it  was 
thought  that  it  would  help  to  reduce  the  total  cost  rapidly.  This 
is  what  participants  were  instructed  to  do,  but  often  neglected. 

Costing-aid  Level  3  provided  the  total  cost  as  well  as  the 
component-part  total  costs,  as  did  Level  2;  in  addition,  it  provided 
a  listing  of  previous  designs,  ordered  according  to  cost,  for  each 
component  part,  i.e.,  for  each  change-record.  This  additional 
information  presented  the  best  design,  and  the  tests  used  to  achieve 
that  design,  for  each  change-record  of  interest  to  the  participant. 

This  is  analogous  to  the  information  available  in  a  non-computer 
system  when  careful,  systematic  records  are  kept  with  paper  and 
pencil  of  each  trial  design.  Such  a  non-computer  system  could  involve 
a  user  talking  with  a  software  expert,  where  the  user  presents 
trial  designs  and  the  expert  computes  the  cost  of  each. 

Although  there  was  a  tendency  for  costing-aid  Levels  2  &  3 
to  degrade  performance  at  problem  Levels  1  ft  2,  and  to  improve 
performance  at  problem  Level  3,  the  statistical  analyses  demonstrated 
that  the  costing-aids  did  not  support  superior,  or  even  satisfactory 
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performance.  Nevertheless,  It  cleanly  is  important  for  a  user  to 
discover  which  features  are  available  at  low  cost  and  to  include 
them  in  the  final  RS,  as  well  as  to  discover  those  features  that 
are  high  cost  and  to  exclude  them.  Consequently,  one  must  consider 
alternative  systems.  Some  of  these  are: 


1.  Train  software  experts  also  to  be  expert  in  an 
application  area,  namely,  the  user's  problem  area. 

This  alternative  is  often  used  now  by  default  -  in 
cases  where  the  user  cannot  develop  RS  or  provides 
incomplete  RS  —  but  is  obviously  the  least  desirable 
alternative  because  the  user  lacks  control  over  the 
product . 

2.  Develop  a  method  for  the  user  to  specify  the  trade-off 
criteria  he  or  she  employs  in  evaluating  alternative  RS 
and  in  selecting  the  final  RS.  With  these  criteria 
specified,  the  RS  could  then  be  constructed  by  the 
software  designer.  This  method  involves  a  totally 
different  concept  from  that  investigated  here  or 
presently  used.  It  will  not  be  investigated  in  this 
project  until  the  third  alternative,  below,  has  been 
investigated . 

3.  Develop  a  method  by  which  the  user  can  develop  more 
complete  and  more  cost-effective  RS.  The  results 
reported  here,  where  the  ability  of  the  managers  to 
develop  cost-effective  RS  was  found  to  be  poor, 
suggest  that  the  new  aids  must  not  only  provide  automatic 
calculations  of  the  least-cost  system,  but  must  also 
guide  the  user  in  collecting  all  the  necessary  data. 

Alternative  #3  is  being  pursued  currently  with  the  design 
of  improved  user-aids . 

Effect  of  Problem-Complexity 

The  ANOVA,  SNK,  and  regression  analyses  showed  that 
problem-complexity  produced  significantly  different  experiment 
results.  The  results  of  the  SNK  analysis  suggest  that  there  was  a 
significant  difference  in  effect  between  problem  Levels  1  and  2,  and 

1  and  3,  but  not  between  Levels  2  and  3.  Future  studies  will 
involve  a  greater  difference  in  problem-complexity  between  Levels 

2  and  3. 


CONCLUSIONS  AND  PLANS 


The  results  suggest  that  the  user/software  interchange 
without  the  benefit  of  sophisticated  costing-aids  does  not  result  in 
RS  capable  of  guaranteeing  a  least-cost  system.  It  is  possible 
that  the  use  of  an  actual  software  expert  instead  of  a  simulated 
costing  expert  might  improve  performance,  i.e.,  that  improvement 
might  have  to  be  directed  by  the  software  expert  rather  left  to 
the  user.  Since  the  goal  here  was  to  develop  and  test  aids  to 
help  a  user  working  with  a  software  expert  in  developing  quality 
RS,  we  conclude  that  sophisticated  aids  must  be  developed  that 
will  assist  in  insuring  that  all  necessary  information  is  obtained 
that  will  provide  the  necessary  calculations  automatically. 
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APPENDIX  A 


Available  Tools/Models/Design  Methods 


Richhart  (1983)  identified  tools,  models,  and  design  methods 
for  RS  and  software  development.  Those  devices  are  listed  in 
this  Appendix. 

Available  Tools/Models/Systems 
Non-Automated  Tools 

HIPO  -  Hierarchical  Input,  Process  &  Output  functional 
design  diagrams  for  modeling  programming  pro¬ 
jects  in  terms  of  levels  of  systems,  programs, 
and  modules.  Used  to  represent  a  system  as  a 
hierarchy  of  input/process/output  modules. 

DFD  -  Data  Flow  Diagrams 

These  indicate  how  data  flows  and  is  transformed 
from  one  kind  of  data  item  into  another  as  it 
passes  through  data-analyzing  systems. 

Nassi-  A  low-level  design  technique  for  graphical  description 
Schneiderman  of  the  structure  and  statements  in  a  class  of  well- 
Diagrams  structured  programs.  (  Nassi,  Schneiderman  1973). 

Structure  -  A  hierarchical  chart  showing  function-modules  and 
Diagrams/  the  functional  flow  and  calling  sequence  between 

Charts  modules.  Some  users  also  include  the  functional 

inputs  and  outputs  for  the  modules  being  called. 

Warmer-  -  Show  an  expansion  of  the  system  on  the  left  side 
Orr  Diagrams  of  the  page  into  increasingly  greater  detail  toward 

the  right  side  of  the  page.  Warnier's  techniques 
have  been  expanded  and  marketed  in  the  U.S.  by 
Ken  Orr  and  Associates.  (Orr,  1991)  (King,  1991). 

Semiautomated  Tools 

Each  of  these  involves  some  form  o*  graphical  notation 
or  formal  language  for  representing  specifications,  as  well  as  a 
systematic  methodology  for  generating  specifications.  (Howden,  1982) 


Interpretive  Models 

PSL/PSA  -  Problem  State  Language/Problem  State  Analyzer. 

Developed  by  the  IS  DOS  project  at  the  University 
of  Michigan.  Specialized  version  of  a  software 
engineering  data  base  for  storing  objects,  object 
properties,  and  relationships  between  objects. 
Employs  both  the  formal  language  of  objects 
and  relations  and  a  graphical  notation  for  displaying 
relationships .  May  result  in  excessive  overhead 
for  small  projects.  (Zelkowitz,  Shaw,  and  Gannon 
1979),  (Teichroew  &  Hershey,  1979) 

HOS/USE  -  Higher  Order  Software  Methodology  by  Higher 

Order  Software  Inc.  of  Cambridge,  Mass.  This 
is  one  of  the  most  complete  and  perhaps  advanced 
of  any  of  the  A  DP  user-oriented  systems.  It 
includes  the  following  automated  tools:  (1)  an 
interactive  graphics  editor  for  entering  and  editing 
HOS  CONTROL  MAPS  (2)  a  mathematically 
based  specification  language  called  AXES 

(3)  Libraries  of  generalized  data  types,  primitives, 
defined  structures,  and  interface  specifications 

(4)  an  Automatic  Analyzer  used  to  detect  specifi¬ 
cation  errors,  (5)  a  Resource  Allocation  Tool 
for  converting  the  analyzed  specifications  from 
the  automatic  analyzer  directly  into  executable 
code,  and  (6)  an  Interactive  Simulator  to  test 
partially  implemented  structures  and  perform 
prototyping.  The  development  tool  that  links  these 
parts  together  and  implements  the  (  methodology 
is  called  USE. IT. 

In  HOS,  the  user  presents  an  analyst  with  a 
statement  of  system  requirements .  The  analyst 
acts  as  a  consultant  to  the  user  by  translating 
his  requirements  interactively  into  functional 
specifications  in  the  form  of  a  hierarchical  control 
map  using  a  simple  graphical-editor  interface. 

Each  node  of  the  hierarchical  diagram  is  specified 
in  terms  of  its  inputs,  process,  outputs,  and 
control  type.  The  analyzer  then  automatically 


converts  the  top-down  hierarchical  control 
structures  from  the  graphical  editor  into  a  formal 
specification  language  called  AXES.  One  of  the 
most  powerful  features  of  this  system  is  its  ability 
to  synthesize  complex  applications  from  libraries 
of  reusable  primitative  operations  that  can  be 
defined  by  the  user.  HOS  is  also  compatible  with 
SADT. 

System  Requirements  Engineering  Methodology,  a 
TRW  requirements-analysis  system  for  real-time 
systems.  Uses  "stimulus  response"  diagrams. 
Consists  of  a  requirements-statement  language 
(RSL)  to  specify  the  relationships  among  the  objects 
of  a  system.  This  system  is  sometimes  called 
the  Software  Requirements  Methodology  (SRM). 
(Zelkowitz,  et  al.  1979) 

Software  [Development  System  by  TRW.  SDS  is 
one  of  the  more  advanced  methodologies  incorporating 
many  different  tools,  including:  Software  Re¬ 
quirements  Engineering  Methodology  (SREM), 
Requirements  Statement  Language  (RSL),  Require¬ 
ments  Engineering  and  Validation  System  (REVS), 
and  Program  Design  Language  2  (PDL2). 

Structured  Analysis  &  Design  Technique  produced 
by  Softech  Inc.  Requirements  representation 
diagrams  (graphical  requirements  language)  uses 
a  manual  graphics  system  for  design  and  analysis. 
Uses  special  forms  for  denoting  system  components, 
their  relationships  with  other  components,  and 
input  and  output  data.  (Ross  &  Schoman,  1977) 
(Zelkowitz,  et  al.  1979) 

Build  Program  Technique  by  John  Rice  &  Olson 
Research  Associates,  BPT  is  the  means  by  which 
the  programmer/analyst  can  build  an  Automatic 
Software  Generation  System  (ASGS).  Specifications 
are  communicated  to  the  build  programs  by  a 
Requirements  Specification  Language  (RSL) 
tailored  to  the  specific  class  of  user.  The  RSL 
statements  are  then  decoded  and  transferred 
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to  the  "build  programs  module"  by  the  RSL  analyzer. 
The  RSL  analyzer  uses  various  skeletal  software 
construction  tailored  for  the  users  and  stored  in 
an  Intermediate  File  (IMF).  (Rice,  1981) 

Prototype  Systems/ Languages 

ACT/1  -  An  IBM  compatible  prototyping  system  developed  by 
Bailey  &  Rose  of  Toronto,  Ontario.  The  essence 
of  the  methodology  is  that  the  system  designer  or 
architect  develops  a  view  of  the  system  based  on 
the  system's  external  description  or  appearance, 
a  view  that  is  both  understandable  and  acceptable 
to  the  users.  The  first  set  of  iterations  makes 
use  of  "scenarios"  constructed  from  sequences  of 
side  play  screens.  This  iteration  process  leads 
to  agreement  on  such  key  matters  as  screen- 
flow-sequences,  screen  content,  and  whether  the 
application  is  to  be  menu-driven  or  forms-driven, 
question  and  answer,  etc.  The  system  specifica¬ 
tions  are  represented  by  a  series  of  machine- 
implemented  application  scenarios,  rather  than 
functional  flowcharts  or  application/structure 
diagrams.  The  developer  concentrates  on  the 
design  and  implementation  of  program  transaction 
modules  that  process  data  between  screens. 

(Mason,  Carey  1 983) 


ADMINIS/11  -  By  Adminis  Corp.  for  PDP/1 1 .  has  graphics 

generator,  report  generator,  and  application 
generator.  Has  on-line  capability  using  its  own 
file  structure,  and  is  suitable  for  end-users. 

DATA  ANALYZER  -  By  Programming  Products  for  IBM  370  systems. 

Has  a  report  generator,  graphics  generator, 

query  language,  and  is  on-line.  Suitable  for 
end-users  and  has  its  own  file  structure. 


FOCU  -  By  Info.  Builders  for  IBM  370.  Has  an  application 
generator,  report  generator,  graphics  generator, 
query  language,  very  high-level  programming 
facilities,  and  is  on-line.  Suitable  for  end-users 
and  has  its  own  file  structure. 


NOMAD  - 


USER/11  - 


PDL  - 


STRUCTURED  - 
DESIGN 


JDM  - 


WARN  IER-ORR  - 
DIAGRAMS 


By  National  C.S.S.  for  N.C.S.S.  Has  a 
relational  data  base  with  a  report  generator, 
graphics  generator,  query  language,  very  high- 
level  programming  facilities,  and  is  on-line. 
Suitable  for  end-users. 

By  Northcounty  for  PDP/1 1 .  Has  an  application 
generator,  a  report  generator,  graphics  generator, 
query  language,  very  high-level  programming 
facilities,  and  is  on-line.  Suitable  for  end-users 
and  has  its  own  file  structure. 


Design  Methods 

Program  Design  Language  (META-languages  for 
precisely  defining  program  modules,  including 
all  data  structures  and  operations  on  the  data). 

Manual  System  for  creating  a  detailed  design, 
consisting  of:  subsystem,  process,  activity,  and 
module.  Includes  development  of  hand-drawn 
baseline  diagram.  (Zelkowitz,  et  al.,  1979) 

Jackson  Design  Methodology  -  based  on  the  principle 
that,  at  the  specification  stage  of  a  program,  the 
element  that  exhibits  the  most  structure  is  that 
of  the  data.  The  Jackson  approach  is  to  define 
the  structure  of  the  data  and  then  to  derive 
the  program  structure  from  the  data  structures. 
Incorporates  the  techniques  of  top-down  develop¬ 
ment,  structured  programming,  and  structured 
walkthroughs.  Programs  are  hierarchically 
structured  with  the  following  four  basic  components: 
elementary,  sequence,  selection,  and  iteration. 

(King,  1981)  (Zelkowitz,  et  al.  1979) 

Warnier’s  approach  includes  three  separate  tech¬ 
niques  known  as:  Logical  Construction  of  Systems 
(LCS),  Logical  Construction  of  Programs  (LCP), 
and  Logical  Construction  of  Execution  (LCE). 
Warnier's  techniques  have  been  expanded  and 
marketed  in  the  U.S.  by  Ken  Orr  and  Associates. 
(Orr,  1981)  (King,  1981) 


INFORMATION  -  Developed  by  INFOCOM,  Australia.  Data  held  in 
ENGINEERING  computer  files  or  data  bases  and  used  to  "model" 

the  organization.  IE  has  an  initial,  data-oriented 
analysis  followed  by  a  procedure-oriented  approach 

DATA-ORIENTED 

1  .  Examine  the  corporate  purpose  and  mission; 
identify  fundamental  data: 

a.  Current  organization  and  mission 

b.  The  direction  the  organization  is  headed. 

c.  The  direction  the  organization  should  be 
headed  in. 

2.  Identify  data  required  for  specific  functional  ar 

3.  Identify  data  needed  for  top  management 
decision  making. 

4.  Data  base  design  to  model  the  organization. 
PROCEDURE-ORIENTED 

1 .  Identify  decision  events  which  bring  about 
data  change. 

2.  Users  develop  formal,  structured-English 
procedure  specifications.  (Finkelstein,  1981) 

DATA  -  Used  for  recording  in  a  centralized  location 
DICTIONARIES  all  decisions  related  to  the  structure  and 

implementation  of  every  element,  record,  and 
file. 


Management  Systems/O'ools 

Manual  Tools 

Milestones 
Gantt  Charts 
Pert 

Critical  Path 


Automated  Management  Systems 


Estimator  -  For  IBM  systems  by  AGS  Management  Systems 
Inc.  An  estimation  tool  for  determining  the  time  and  cost  estimates 
for  each  phase  of  a  system-development  life  cycle. 

G/C  CUE  -  For  H-P  3000,  Prime,  Dec  &  Vax  Equipment  by 
Gilbert/Commonwealth.  Provides  estimating,  accounting,  planning, 
scheduling,  and  cost-performance  measurement  information. 

Spectrum-3-  For  IBM  &  Apple  computers  by  Spectrum 
International  Inc.  A  project  estimating  tool  which  provided 
estimated  guidelines  in  terms  of  person  hours,  cost,  and  schedule 
of  the  project  work. 

CSSR  -  Runs  on  H-P  3000  Systems,  produced  by  AGS 
Management  Systems,  Inc.  Monitors  and  forecasts  cost  and 
schedule  performance  on  smaller  acquisitions/projects. 

PAC  II/III  -  Runs  on  IBM,  DEC  VAX,  and  System  10-20 
computers,  by  AGS  Management  Systems,  Inc.  Provides  project 
management  for  large  projects.  Combines  cost, time  and  resource 
factors  to  forecast  when  each  project  activity  will  be  completed, 
how  and  at  what  cost.  Calculates  critical  paths,  resource 
bottlenecks,  and  keeps  detailed  cost/accounting  information. 

PC/70  -  Runs  on  IBM  and  HP  3000  Systems,  by  AGS 
Management  Systems,  Inc.  Designed  to  forecast  schedules, 
cost-and  workloads;  pinpoint  trouble  spots;  simulate  schedules, 
measure  performance  and  progress . 

N5500  Project  Planning  and  Control  System  -  Runs  or,  IBM, 
DEC,  Burroughs,  HP  3000,  Honeywell,  CDC,  UNIVAC,  Prime, 
WANG.  Built  by  Nicholas  &  Company,  determines  project  trends 
and  predicts  completion  dates  &  costs. 

Prompt  Aid  1  (for  estimator)  -  Runs  on  an  Intertec 
Superbrain,  produced  by  Simpact  Systems.  Assists  users  in 
determining  the  expected  cost  and  effort  of  a  computer  development 
project. 


Richhart  (1983)  gives  the  following  description  of  RS 
development  and  use  in  the  DOD: 

The  U.S.  Military  has  recognized  the  need  to  increase  the 
amount  of  attention  given  to  identifying  user  needs  and  to  developing 
requirements  specifications.  The  U.S.  Navy  has  implemented  DOD 
guidance  for  developing  an  ADP  system  through  the  use  of  NAVDAC 
PUB  24.1  and  24.2.  The  U.S.  Air  Force  has  implemented  this 
guidance  in  Air  Force  regulation  AFR  300-12  and  AFR  300-15. 

Prior  to  the  development  of  System  Specifications  (SS)  in 
the  definition  phase,  both  the  Navy  and  the  Air  Force  require  a 
mission-analysis  phase  and  a  concept-development  phase.  The 
mission-analysis  phase  is  intended  as  a  means  for  extracting  and 
identifying  the  essential  needs  of  the  user,  recognizing  that  the 
user  may  not  know  exactly  what  the  problem  is  or  how  ADP 
resources  might  best  be  able  to  help.  In  the  Navy,  this  phase 
is  accompanied  by  the  development  of  a  Mission  Element  Need 
Statement  (MENS)  and  a  statement  of  General  Functional  Re¬ 
quirements  (GFR).  In  the  Air  Force,  a  feasibility  study  is 
performed  in  conjunction  with  the  identification  of  the  existing 
functional  baseline. 

Both  services  follow  their  initial  investigation  with  a 
conceptual  phase  to  develop  the  initial  Concept  of  Operations 
(CONOPS)  and  an  economic  analysis  (USAF)  or  System  Decision 
Paper  (SDP)  (USN)  to  determine  whether  further  development 
effort  is  justified.  If  a  decision  is  made  to  proceed,  a  detailed 
Functional  Description  (FD)  is  prepared  describing  the  functions 
to  be  developed  or  integrated  with  a  new  system.  In  the  Navy, 
a  Plan  of  Action  and  Milestones  (POA  &  M)  document  is  prepared. 
The  Air  Force  equivalent  of  the  POA  &  M  is  the  Data  Project 
Plan  (DPP).  When  all  of  this  documentation  is  ready,  and  the 
users  have  had  a  chance  to  review  it,  a  System  Requirements 
Review  (SRR)  is  held  with  key  representatives  from  all  of  the  user 
and  ADP  offices  attending.  The  SRR  is  a  rather  painstaking, 
drawn  out  review  that  frequently  proceeds  paragraph  by  paragraph 
through  the  FD  and  milestones.  It  is  not  at  all  unusual  for  the 
users  to  ask  so  many  questions  or  make  so  many  corrections  that 
a  follow-up  SRR  is  required  to  resolve  the  differences.  When 
the  SRR  is  finished  and  the  corrected  FD  and  milestones  are 
approved  by  the  user,  they  form  a  new  functional  baseline. 
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In  the  system-design  phase,  the  detailed  System  Specifications 
(SS)  are  developed  along  with  a  Data  Requirements  Document  (DRD), 
and  an  Interface  Control  Document  (ICD).  These  documents  are 
reviewed  by  the  users  and  approved  in  a  System  Design  Review 
(SDR)  which  is  very  similar  to  the  SRR.  The  SDR  establishes  a 
new  design  baseline  for  the  hardware,  communications,  and 
software  (called  an  "allocated  baseline"  in  the  USAF). 

The  SDR  is  followed  by  a  more  detailed  subsystem-design 
and  data-base  specification  that  is  finalized  in  a  Preliminary  Design 
Review  (PDR).  In  smaller  systems,  the  PDR  and  SDR  are  often 
combined  into  one  larger  SDR.  Each  program  then  goes  through  a 
design  phase  and  the  development  of  detailed  Program  Specifications 
(PS)  which  are  approved  in  small  Critical  Design  Reviews  (CDR) 
between  the  responsible  programmer,  his/her  manager,  and  the 
individual  user  who  will  eventually  use  the  program. 

All  of  the  preceeding  work  is  done  before  the  System 
Development  Phase  gets  the  developer  involved  with  actual  pro¬ 
gramming  of  the  individual  modules.  While  this  looks  okay  on 
paper,  obviously  there  is  a  lot  of  overhead  involved  with  this 
type  of  specification  and  design.  To  shorten  the  process  so 
that  small  projects  are  not  hampered,  thresholds  are  used  to 
select  the  projects  that  must  follow  the  full  development  schedule. 
Even  when  the  size  of  a  project  should  require  it  to  follow  these 
guidelines,  it  is  often  excused  because  of  an  operational  necessity 
to  meet  operational  dates. 
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The  matrices  in  Figures  Cl  -  C7  are  the  costing  matrices 
described  in  the  section  "Method  of  Approach:  The  Participant's 
Task.  ” 
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50.00,50.00,50.00,2.50,50.00,50.00,2.50,50.00,5.00,25.00,50.00 

50.00,5.00,50.00,5.00,50.00,7.50,50.00,7.50,5.00,7.50,5.00 

50.00,50.00,5.00,7.50,50.00,50.00,5.00,7.50,50.00,50.00,5.00 

7.50,50.00,50.00,5.00,7.50,50.00,50.00,50.00,7.50,50.00,50.00 


Note:  This  is  a  sequential  4x11  array  beginning  with  i=1,  j=1 
and  ending  with  i=14,  j=4.  The  index  j  takes  on  the 
maximum  value  of  2,  3,  or  4  depending  on  the  value  of  i. 


See  Figure  C3  for  the  full  run  of  i  and  j. 
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Note:  This  is  a  sequential  4x11  array  beginning  with  i=1,  j=1 
a  and  ending  with  i=14,  j=4.  The  index  j  takes  on  the 

maximum  value  of  2,  3,  or  4  depending  on  the  value  of  i. 
See  Figure  C3  for  the  full  run  of  i  and  j. 
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Newspaper  Ad  placed  in  the  Washington  Post  on  May  1,  1983, 

May  8,  1983,  and  June  12,  1983: 

MANAGEMENT:  A  firm  in  Vienna  is  seeking  the  help  of  36 
managers  with  inventory  control  experience  to  evaluate  a  computer 
system.  This  work  is  sponsored  by  the  Department  of  Navy. 

The  computer  evaluation  will  take  approximately  4  1/2  hours 
(1  day  only).  Qualifications:  Degree  in  Business  Management  or 
4  years  management  experience,  including  inventory  control. 

$12. 00/hr.  Call  Susan  for  details.  9  a.m.  to  1  p.m.  Mon- 
Fri.  938-1603. 


Participant  No 


Inventory  Control  Systems : 


(D 


(3) 


(5) _ 

Please  make  any  additional  comments  you  believe  best  describes  your 
profess  ional  expe  r  ience . 
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Higher  Level  Language  Programming  Experience 

Please  check  all  languages  in  which  you  have  programmed  and  list 
your  best  estimate  of  the  number  of  programs  coded  in  that  language. 

TYPE  NUMBER  OF  PROGRAMS  CODED 

BASIC 

_  FORTRAN 

_  COBOL 

_  PL-1 

_  ALGOL 

PASCAL 


Enter  your  best  estimate  of  the  total  number  of  programs 
you  have  coded  (any  language,  any  computer).  _ 


Data  Processing  Experience 

Enter  the  number  of  months  of  full-time  experience  you  have  in  thu  following: 
Number  of  Months 

_  Data  Entry 

_  Production  Control 

_  Operations 

_  Applications  Programming 

_  System  Programming 

_  System  Analysis 

_  Data  Base  Administration 

_  Data  Communications 

_  Other(s) 


Please  give  the  number  of  year's  you  have  worked  in  the  computer 
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Enter  your  best  estimate  of  the  number  of  programs  for 
which  you  have  coded  the  Job  Control  Language  (JCL)  necessary 
for  testing  or  production  runs.  _ 


Please  list  up  to  5  computer/operating  systems  on  which 
you  have  worked  which  you  believe  best  represents  your  experience. 
Examples:  IBM  370/155  OS,  PDP-11  RT-11. 


Specifications  of  Data  Base  Systems 

Please  list  the  number  of  Data  Base  Systems  including 
inventory  systems  you  have  used:  For  each  indicate  if  you  are 
an  user,  specific  or  designer. 

Data  Base  System  User  Specific  Designer 


Please  enter  the  number  of  years  experience  with  Data  Base  Systems 


CONTRACT  TO  ACT  AS  A  RESEARCH  PARTICIPANT 

FOR 

THE  DEPARTMENT  OF  THE  NAVY 


Name  of  Participant:  Date: 

Address: _ 

Street  City  State  Zip  Code 


1*  I  authorize  Mr.  Edward  M.  Connelly,  who  is  the  Principal  Investigator 
for  this  project,  or  his  representative,  to  collect  and  analyze  my 
solutions  to  test  problems. 

2.  This  project  has  been  explained  to  me  by _ 

(Print) 

It  has  been  pointed  out  to  me  that  my  solutions  will  not  be  made 
known  to  any  Individual,  other  than  appropriate  members  of  the 
research  team,  or  for  any  purpose  other  than  the  data  analysis 
to  be  performed  by  the  research  team.  No  Information  concerning 
my  performance  on  the  experimental  task  will  be  disclosed  to  any¬ 
one  other  than  the  research  team  without  my  written  permission. 

3.  I  understand  that  there  are  no  special  risks  of  any  kind  associated 
with  my  participation  in  this  study. 

4.  I  understand  that  the  project  may  further  the  understanding  of  how 
various  aids  can  assist  a  programmer  or  other  person  to  specify  a 
computer  program. 

5.  I  understand  that  Mr.  Edward  M.  Connelly  or  Joanne  C.  Connelly 
will  answer  arty  questions  that  I  may  have  about  this  project. 

6.  I  understand  that  by  signing  this  form,  I  have  waived  none  of  my 
legal  rights  that  may  be  associated  with  liability  for  negligence  on 
the  part  of  Performance  Measurement  Associates,  Inc. 

7.  I  state  that  I  am  18  years  of  age  or  older. 


(Participant  Signature) 


(Adm  inlstrator) 
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